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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  research  reported  here  is  part  of  a  broader  program  on  training 
for  combat  effectiveness  being  conducted  by  the  US  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences.  The  availability  of  current  and 
sufficient  knowledge  of  troop  combat  readiness  and  existing  skill  defi¬ 
ciencies  is  critical  to  resource  allocation  and  training  management  in 
units.  The  ARl  Field  Unit  at  Fort  Banning,  Georgia,  is  developing  the 
methods  and  guidelines  necessary  to  Implement  a  partial  answer  to  this 
need:  the  design  and  use  of  unit  training  devices  to  satisfy  personnel 
evaluation  and  qualification  purposes  also.  This  report  describes  an 
analytical  method  for  assessing  the  cost  effectiveness  of  using  a  partic¬ 
ular  device  or  any  one  of  a  set  of  alternative  devices,  and  how  often  to 
use  it,  for  evaluation  of  personnel  combat  readiness  and  skill  deficiency. 

The  method  will  be  useful  to  training  developers  (e.g.  USAIS  DTD)  in 
deciding  whether  or  not  to  require  a  capability  on  the  part  of  a  training 
device  to  evaluate  personnel  performance  deficiencies  and  qualifications. 
The  method  will  be  useful  to  device  developers  (e.g.  PM  TRADE)  in  deciding 
which  training  device  would  best  Implement  the  requirement  from,  both  cost 
and  benefit  standpoints. 
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COST  AND  INFORMATION  EFFECTIVENESS  ANALYSIS  (CIEA) :  A  METHODOLOGY  FOR 
EVALUATING  TRAINING  DEVICE  OPERATIONAL  READINESS  ASSESSMENT  CAPABILITY 
(DORAC) 


BRIEF 


Requirement ; 

Up  to  date  information  on  the  combat  readiness  and  skill  deficiencies 
of  troops  is  needed  to  report  personnel  readiness  levels  and  to  manage 
troop  training.  Meeting  this  need  can  be  made  difficult  by  frequent 
personnel  turnover;  infrequent  testing;  and  unknowns  regarding  skill  decay 
rates.  A  partial  solution  to  this  problem  may  reside  in  unit  training 
devices  that  are  currently  fielded  and/or  under  development.  If  a  device 
is  designed  and  used  to  satisfy  training  requirements,  it  may,  in  many 
cases,  also  be  designed  and  used  on  a  regular  basis  (e.g.,  quarterly)  to 
determine  personnel  qualification  status  and  skill  deficiencies  requiring 
training.  However,  the  Inclusion  of  the  evaluation  capability  may,  some¬ 
times,  be  more  expensive  than  the  payoff  can  justify.  This  calls  for  a 
cost  benefits  analysis  method  to  assess  whether  a  particular  device,  or  one 
of  a  set  of  device  design  alternatives,  is  worth  using  in  this  manner. 

I.e.,  a  method  to  determine  if  the  information  that  can  be  gained  from 
evaluation  of  performance  on  the  device  is  worth  the  costs  of  the  evaluation 
process  and  any  necessary  modifications  to  the  device  design  (e.g.,  incor¬ 
poration  of  an  expanded  visual  simulation  capability  or  a  performance 
measurement  software  system). 


Procedure! 

A  review  of  the  literature  on  cost  benefits,  information  theory,  and 
utility  evaluation  was  conducted  to  identify  and  examine  current  methods. 
Existing  models  and  techniques  were  integrated  and  further  developed  into  a 
multiattribute  utility  measurement  (MAUM)  Cost  and  Information  Effectiveness 
Analysis  (CIEA)  method.  The  method  was  tested  and  further  refined  through 
application  to  a  set  of  testing  alternatives  for  M16A1  rifle  marksmanship 
skills.  The  alternatives  included  the  Weaponeer,  Squad  Weapon  Analytical 
Trainer  (SWAT),  and  the  current  field  testing  procedure.  Record  Fire.  This 
report  presents  the  details  of  the  developed  MAUM  CIEA  method  and  presents 
fictitious  data  for  illustration  purposes.  The  details  of  the  application 
and  actual  resulting  data  are  presented  in  a  companion  report  (Hawley  & 
Dawdy,  1981). 
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Findings: 


(1)  The  developed  MAUM  CIEA  procedures  were  applied  successfully  in  an 
evaluation  of  alternatives  for  testing  M16A1  rifle  marksmanship  skills.  Army 
officers  with  training  device  expertise,  and  other  officers  with  operational 
unit  responsibilities,  were  able  to  apply  the  MAUM  CIEA  to  the  evaluation  and 
the  resulting  values  appeared  reasonable. 

(2)  There  is  a  need,  however,  to  simplify  and  expedite  the  application 
of  the  MAUM  CIEA  method  to  the  extent  possible.  Possible  avenues  of  approach 
will  be  explored  in  future  research. 


Utilization  of  Findings: 

The  ultimate  users  of  the  MAUM  CIEA  method  will  be  US  Army  Training  and 
Doctrine  Command  (TRADOC)  training  developers  and  device  design  engineers 
working  with  the  materiel  developer,  PM  TRADE.  The  method  will  be  used  to 
determine  if  and  how  to  develop  and  Implement  DORAC  in  existing  training 
devices  and  in  devices  under  development.  Further  work  will  be  done  to 
simplify  and  validate  the  method  to  make  it  practical  for  use  by  these 
personnel. 
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SECTION  1 


INTRODUCTION 

Background 


Most  scenarios  for  a  full-scale  confrontation  between  the  United  States 
and  any  of  its  major  potential  adversaries  indicate  that  the  majority  of 
Army  units  will  have  to  be  prepared  to  fight  immediately  without  the  luxury 
of  a  lengthy  mobilization  period,  such  as  the  first  year  of  World  War  II. 
Studies  of  the  comparative  military  strengths  of  the  United  States  and  the 
Warsaw  Pact  countries  also  indicate  that  U.S,  forces  are  likely  to  be  heav¬ 
ily  outnumbered,  often  by  a  ratio  of  five  to  one  or  more.  To  have  any  hope 
of  success  in  this  "come  as  you  are  and  win  the  first  battle  outnumbered" 
situation,  the  Army  will  have  to  maintain  a  high  level  of  individual  and 
unit  combat  readiness  at  all  times.  Maintaining  a  consistently  high  level 
of  combat  readiness  will  necessitate  frequent  evaluations  of  individual  and 
unit  proficiency,  along  with  a  means  of  quickly  diagnosing  and  remediating 
performance  problems. 

In  simpler  times,  the  assessment  of  individual  and  unit  job  proficiency 
presented  no  special  difficulties.  Recently,  however,  the  complex  nature 
of  many  weapons  systems,  personnel  turnover  rates,  and  changes  in  training 
philosophy  have  led  to  an  emphasis  on  performance-oriented  training  and  cri¬ 
terion-referenced  testing.  Soldiers  are  required  to  demonstrate  individual 
and  collective  competencies  in  a  "hands  on"  environment  using  actual  equip¬ 
ment.  This  change  in  training  and  evaluation  methodology  has  increased  the 
demand  for  training/evaluation  uses  of  operational  equipment  and  accompany¬ 
ing  support  resources  requirements  [e.g.,  ammunition,  spare  parts,  POL 
(petroleum,  oil,  and  lubricants),  etc.]  during  a  time  of  inflation  and 
budgetary  constraints.  Mora  frequent  performance-oriented  individual  or 
collective  readiness  evaluations  would  tend  to  complicate  this  already  tight 
situation. 

A  proposed  solution  to  the  problem  of  conducting  more  frequent  readi¬ 
ness  evaluations  in  the  face  of  tight  resource  constraints  is  to  use  train¬ 
ing  devices  instead  of  actual  equipment  in  the  conduct  of  these  evaluations 
(Hopkins,  1975).  In  addition  to  their  uses  in  training,  training  devices 
(e.g.,  simulators,  mock-ups,  etc.)  can  often  provxde  a  vehicle  for  indi¬ 
vidual  or  collective  proficiency  evaluations  (Fitzpatrick  &  Morrison,  1971; 
Glaser  &  Klaus,  1972;  Finley,  Gainer,  &  Muckier,  1974;  Crawford  &  Brock, 
1977).  Historically,  th  '  most  extensive  use  of  training  devices  in  pro¬ 
ficiency  assessment  has  been  in  aviation  (Caro,  1973).  The  commercial  air¬ 
lines  and  the  Federal  Aviation  Administration  use  flight  simulators  exten¬ 
sively  in  proficiency  assessment.  Follow-up  studies  have  indicated  that 
pilot  performance  in  flight  simulators  is  predictive  of  performance  in  ac¬ 
tual  aircraft  (American  Airlines,  1969;  Weltzman,  Fineberg,  Gade,  &  Compton, 
1979). 
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Within  a  military  setting,  the  uses  of  training  devices  in  performance 
evaluation  have  generally  mirrored  civilian  uses  and  primarily  involved 
aviation.  Ttiere  has  been,  however,  an  increasing  use  of  training  devices 
to  assess  individual  and  collective  proficiency  in  other  areas.  Among  the 
other  applications  have  been  maintenance  performance  and  anti-submarine 
warfare  (ASW)  crew  proficiency.  In  the  Army,  one  long-standing  program  of 
individual  and  collective  readiness  assessment  using  a  training  device  is 
found  in  the  Air  Defense  branch.  Here,  the  AN/TPQ-29  simulator  (and  prior 
to  that  the  AN/MPQ-Tl  simulator  for  the  Nike-Hercules  system)  is  used  in 
the  conduct  of  operation  readiness  evaluations  for  HAV'/K  Air  Defense  units. 
The  AN/TPQ-29  is  an  engagement  simulator  capable  of  generating  a  variety 
of  simulated  air  defense  combat  situations  [e.g.,  multiple  targets,  elec¬ 
tronic  countermeasures  (ECM)  of  various  kinds,  etc.].  The  simulator  was 
designed  primarily  for  use  as  a  training  device,  but  it  can  be  (and  is) 
used  to  evaluate  individual  and  crew  performance.  Wlnen  using  the  AN/TPQ-29 
in  assessment,  an  evaluation  team  loads  a  "raid  tape"  containing  the  param¬ 
eters  of  an  air  defense  engagement  into  the  simulator.  The  HAWK  crew  is 
evaluated  on  its  ability  to  defeat  the  simulated  threat;  performance  check¬ 
lists  are  used  to  evaluate  individual  crew  member  performances.  Hardcopy 
printouts  of  some  individual  and  crew  performance  measures  (e.g.,  targets 
destroyed,  numbers  of  penetrators,  engagement  times,  operator  reaction 
times,  etc.)  are  also  obtainable  from  the  simulator. 

The  evaluation  of  HAWK  personnel  using  the  AN/TPQ-29  engagement  simu¬ 
lator  illustrates  the  concept  of  a  Training  ^evice  Operational  Readiness 
Assessment  O^P^bility  (DORAC) .  DORAC  simply  means  that  an  operational 
readiness  assessment  capability  is  included  with  the  training  devices  for 
a  materiel  system.  Once  built  into  the  training  device  system,  the  DORAC 
can  be  used  to  assess  the  operational  readiness  of  individuals  or  crews. 

As  an  added  feature,  the  measurement  capability  inherent  in  a  DORAC  can 
also  provide  information  useful  for  other  purposes  such  as  training  manage¬ 
ment,  unit  management,  materiel  system  evaluation,  and  the  like. 

A  recent  review  of  Army  training  device  proficiency  assessment  capa¬ 
bilities  has  indicated  that  the  DORAC  principle  can  be  applied  to  the  train¬ 
ing  devices  for  virtually  any  parent  materiel  system  (Shelnutt,  Smillie, 

&  Bercos,  1978).  At  the  present  time,  actual  use  as  in  the  HAWK  system 
and  in  the  aviation  community  is  somewhat  rare,  but  the  potential  remains. 
There  are,  however,  several  issues  that  remain  to  be  resolved  before  gen¬ 
erally  attempting  to  implement  the  DORAC  principle  v/ithin  the  Army.  Among 
these  issues  are  the  feasibility  of  the  concept,  the  utilit.  of  the  result¬ 
ing  information,  and  a  means  for  determining  what  performance  data  are  ac¬ 
tually  required.  One  of  the  primary  issues  involves  a  workable  methodology 
for  establishing  the  cost-effectiveness  of  a  DORAC;  that  is,  determining 
the  conditions  under  which  the  information  (readiness,  training  status, 
etc.)  derived  from  a  particular  DORAC  is  worth  the  cost  required  to  obtain 
it.  This  determination  may  be  needed  as  a  part  of  specifying  the  require¬ 
ments  for  a  device  or  as  a  basis  for  deciding  betveen  two  or  more  design 
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options  developed  to  satisfy  the  requirements.  The  objective  of  this  re¬ 
port  is  to  present  a  methodology,  denoted  Cost  and  Information  Effective¬ 
ness  Analysis  (CIEA) ,  for  evaluating  DORAC  alternatives  in  terms  of  the 
value  of  the  information  provided  versus  the  cost  of  developing,  imple¬ 
menting,  and  operating  the  capability.  The  alternatives  include  a  base¬ 
line  case,  i.e.  ,  the  current  operational  readiness  procedure  and  frequency 
(e.g.,  an  annual  ARTEP) ;  and  one  or  mote  training  device  designs  operated 
at  one  or  more  annual  frequencies  for  evaluation  purposes.  The  costs  of 
developing  and  operating  the  device  to  satisfy  training  requirements  are 
not  included  in  the  DORAC  CIEA;  the  only  costs  included  are  those  asso¬ 
ciated  with  a  simulation  or  measurement  capability,  or  evaluation  processes 
(e.g.,  administration  support  costs),  which  are  over  and  above  those  re¬ 
quired  to  support  training  requirements  per  se.  CIEA  is  intended  to  serve 
as  a  guide  for  decision-makers  in  deciding  whether  DORAC  should  be  included 
in  any  training  device,  irrespective  of  the  parent  materiel  system. 


Assessing  Information  Worth 


The  ultimate  objective  of  CIEA  is  to  facilitate  assessing  the  relative 
worth  of  the  differential  information  obtained  from  alternative  DORACs  that 
have  different  costs  and  other  associated  resource  requirements.  Informa¬ 
tion  (readiness,  training  status,  etc.)  has  value  only  when  it  results  in 
a  gain  to  the  receiving  party;  thus,  information  should  be  collected  only 
when  the  incremental  cost  of  obtaining  it  is  less  than  the  gain  that  is 
realized  through  its  receipt.  CIEA  is  a  procedure  for  determining  whether 
or  which  DORAC  alternatives  are  worth  implementing.  Estimating  the  cost 
required  to  obtain  a  given  amount  of  information  is  a  relatively  straight¬ 
forward  procedure.  However,  determining  the  value  of  the  same  quantity  of 
information  is  not  as  straightforward.  A  given  piece  of  information  may 
have  value  for  one  purpose  but  not  for  another.  Information  value  depends 
on  a  particular  communication's  relevance  for  potential  action  by  a  spe¬ 
cific  set  of  decision  makers  acting  in  a  specified  environment  at  a  certain 
point  in  time. 

Before  addressing  the  problem  of  assessing  information  worth,  it  is 
first  necessary  to  define  what  is  meant  by  the  term  information.  Bedford 
and  Onsl  (1966)  define  inforvnation  as  "data  evaluated  for  a  specific  use." 
Reviewing  information  is  characterized  as  a  process  of  ignorance  reduction. 
The  function  of  information  is  to  reduce  the  amount  and  range  of  uncertainty 
under  which  decisions  are  made. 

Two  attributes  are  associated  with  information:  Amount  and  Value. 

The  amount  of  information  in  a  communication  is  determined  by  the  reduction 
in  uncertainty  resulting  from  its  receipt.  Amount  is  assessed  formally 
through  the  application  of  measures  such  as  Shannon's  H  (Shannon,  1948; 
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Shannon  &  Weaver,  1963).  Information  value  is  a  function  of  the  cruvijos 
resulting  from  the  use  of  information  in  the  pursuit  of  a  particular  pur¬ 
pose  (Bedford  &  Onsi,  1966).  A  communication  may  contain  a  large  amount 
of  information  without  being  valuable  in  the  sense  of  saying  something 
useful  to  the  recipient.  Information  value  is  measured  by  a  receiver  in 
terms  of  the  information's  uses  in  decision  making.  Strictly  speaking, 
this  definition  implies  that  Information  value  is  measured  by  comparing 
the  actions  of  decision-makers  before  and  after  the  receipt  of  a  given 
quantity  of  information.  Value  is  determined  from  the  gain  realized  as 
the  result  of  taking  one  course  of  action  versus  another  after  receiving 
a  given  amount  of  information  (Bedford  &  Onsi,  1966;  Thiel,  1967;  Lev, 
1969).  In  the  present  situation,  this  view  requires  that  the  value  of 
DORAC  information  strictly  be  measured  by  assessing  the  gain  associated 
with  an  information  recipient’s  taking  one  course  of  action  versus  another 
on  the  basis  of  having  received  some  amount  of  information;  for  example, 
by  training  for  task  X  instead  of  for  task  Y,  or  by  training  P  hours  this 
week  instead  of  Q  hours. 

Determining  information  value  through  a  decision-maker  action  substi¬ 
tution  approach  as  described  above  would  seem  to  be  a  relatively  straight¬ 
forward  procedure.  However,  some  serious  potential  problems  are  apparent. 
First  of  all,  the  logistics  of  such  an  information  worth  evaluation  pro¬ 
cedure  would  likely  be  unmanageable.  For  example,  the  alternatives  of  a 
range  of  decision  makers  would  have  to  be  considered,  the  differential 
costs  of  numerous  before  and  after  decision  scenarios  would  have  to  be 
determined,  the  costs  associated  with  some  action  differentials  might  be 
difficult  or  impossible  to  quantify,  and  so  on.  Based  on  a  preliminary 
review  of  such  potential  problems,  it  is  doubtful  that  an  action  substitu¬ 
tion  approach  to  information  worth  assessment  would  prove  to  be  a  workable 
or  useful  approach  to  the  conduct  of  CIEA. 

An  alternative  worth  evaluation  approach  that  holds  promise  in  the 
current  problem  situation  is  to  consider  information  utility  rather  than 
information  value  in  its  strict  sense.  Utility  is  defined  as  "Psycho¬ 
logical  value"  (Johnson  &  Huber,  1977).  Information  utility  is  thus  de¬ 
fined  as  "the  psychological  or  subjective  value  of  information  in  the  pur¬ 
suit  of  a  particular  purpose."  The  use  of  information  utility  as  a  proxy 
measure  for  information  value  assumes  that  high  information  utility  repre¬ 
sents  at  least  a  propensity  to  make  decisions  as  the  result  of  the  receipt 
of  so.’ne  given  piece  of  information.  An  added  assumption  is  that  informa¬ 
tion  value  (IV)  is  a  monotonically  increasing  function  of  information 
utility  (lU);  that  is. 


IV.  =  $  (lU.) 

1  '  1 

where  IV,  <  IV.  whenever  lU,  <  lU . . 

i  -  J  i  -  J 


(1-1) 
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The  use  of  information  utility  as  a  proxy  for  information  value  re¬ 
duces  the  problem  of  measuring  information  value  to  the  task  of  assessing 
the  utility  of  specific  categories  of  information  for  specific  users.  Since 
a  well  developed  technology  is  associated  with  utility  measurement,  assess¬ 
ing  information  utility  constitutes  a  more  manageable  task  than  assessing 
strict  information  value.  The  next  portion  of  the  report  presents  an  over¬ 
view  of  utility  theory  and  utility  measurement  procedures  as  they  relate  to 
the  issue  of  information  worth  assessment  in  CIE.'.. 


Utility  Measurement 


Theory 

The  branch  of  utility  theory  that  is  most  applicable  to  the  CIEA  prob¬ 
lem  is  termed  riskless  multiattribute  utility  measurement  (MAUM) .  The 
basic  idea  underlying  MAUM  is  quite  simple  (for  example,  see  Raiffa,  1968; 
Keeney  &  Raiffa,  1976):  Every  outcome  of  an  action  has  a  value  or  utility 
on  a  number  of  different  attributes,  dimensions,  or  factors.  The  objective 
of  MAUM,  in  any  of  its  numerous  versions,  is  to  determine  these  values, 
one  factor  at  a  time,  and  then  to  combine  them  across  factors  using  a  suit¬ 
able  aggregation  rule. 

To  introduce  the  notation  and  concepts  of  MAUM,  let  be  a  particular 
value  of  factor  Furthermore,  let  X^,  where  e  X^,  represent^^the  set 
of  all  possible  values  taken  on  by  factor  For  example,  the  i"^  factor 
could  be  "color",  which  might  be  one  of  several  attributes  considered  when 
buying  a  car.  Let  there  be  a  total  of  ^  factors  under  consideration  (e.g., 
price,  style,  economy,  etc.).  Any  given  alternative  is  represented  as  a 
particular  attribute  combination: 

f ,  ^2 ’  •  •  •  >  •  (1—2) 

The  tendency  to  prefer  one  alternative  over  another,  say  a  Brand  A  to 
Brand  B,  is  represented  by  a  construct  called  utility,  denoted  U. 

In  the  MAUM  context,  it  is  assumed  that  the  utility  of  an  alternative 
is  a  function  of  the  utility  of  the  individual  attributes;  that  is, 

U  =  F(x. ,  X,,  X.,  ...,  X  ).  (1-3) 

1  z  ^  n 

Furthermore,  it  is  assumed  that  there  exists  a  utility  measure  for  each 
value  of  each  factor,  denoted  u(x^).  What  MAUM  seeks  is  a  functional  re¬ 
lationship  among  the  Individual  factor  utilities, 

U  =  F[u(Xj^),  u(x2),  u(x^),  ...,  u(x^)],  (1-4) 

that  defines  an  explicit  value  structure  and  can  serve  as  a  basis  for  deci¬ 
sions  concerning  alternatives. 
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Application 


The  application 
out  in  three  phases, 


of  MAUM  in  the  evaluation  of  alternatives  is  carried 
namely : 


1.  Problem  structuring 

2.  Utility  assessment 

3.  Utility  synthesis 


Each  of  the  tliree  phases  is  discussed  in  the  following  paragraphs. 


Problem  Structuring.  Phases  two  and  three  of  the  MAUM  process  provide 
the  tools  for  evaluating  various  alternatives.  The  tools  are  of  limited 
value,  however,  unless  they  are  applied  to  an  appropriately  structured  prob¬ 
lem.  Structuring  a  utility  measurement  problem  is  the  most  critical  aspect 
of  MAUM.  Problem  structuring  is  also  the  phase  that  is  least  amenable  to 
formal  methodological  treatment.  There  are  no  clear-cut  procedures  that 
guide  an  analyst  to  a  correct  problem  structure. 

Research  has  slio\^m  that  the  complexity  of  MAUM  applications  is  in¬ 
versely  related  to  the  effort  spent  in  problem  structuring  (Edwards,  1971; 
Keeney  &  Raiffa,  1976;  Johnson  &  Huber,  1977).  As  a  rule,  the  better  the 
problem  is  defined,  the  simpler  the  MAUM  process  becomes,  and  the  more 
usable  and  acceptable  the  resulting  solution.  In  terms  of  the  attributes 
of  a  correct  problem  structure,  Johnson  and  Huber  (1977)  specify  three 
characteristics,  listed  as  follo'ws: 

1.  The  questions  and  issues  underlying  the  problem  situation 
are  isolated  and  specified. 

2.  The  context  within  which  utility  is  to  be  evaluated  is 
well  defined. 

3.  The  attributes  or  factors  that  are  involved  in  the 
analysis  are  well  defined. 

Detailed  discussions  of  the  heuristics  of  problem  structuring  in  MAUM  are 
presented  in  Edwards  (1971),  Edwards  (1976),  Keeney  and  Raiffa  (1976),  or 
Johnson  and  Huber  (1977). 

Utility  Assessment.  Once  an  assessment  problem,  has  been  adequately 
structured,  the  second  phase  of  MAUM  involves  determining  the  partial  util¬ 
ities  V.  f  each  alternative  on  each  factor.  The  procedures  that  are  used  to 
determine  individual  factor  partial  utility  scores  are  all  direct  exten¬ 
sions  of  psychological  scaling  methods  [see  Guilford  (1954)  ,  Torgerson 
(1958),  Coombs  (1964),  or  Stevens  (1975)].  These  scaling  procedures  are 
classified  into  five  categories,  listed  as  follows: 


1.  Ranking  procedures 

2.  Category  methods 

3.  Direct  methods 

4.  Gamble  methods 

5.  Indifference  methods 

Ranking  methods  require  ordering  the  levels  of  a  factor  from  most 
preferred  to  least  preferred.  These  methods  are  very  easy  to  use;  they 
also  have  been  found  to  produce  results  that  are  acceptable  to  decision 
makers.  The  resulting  scale  values  are  interpreted  as  ordinal  measures 
of  the  relative  worth  of  each  of  the  alternatives  on  the  factor  in  ques¬ 
tion.  Ranking  methods  are  not  often  used  as  a  sole  basis  for  utility 
assessment.  They  are,  however,  frequently  used  as  a  first  step  in  utility 
evaluation,  after  which  another  method  (e.g.,  a  Direct  method)  is  used  to 
obtain  quantitative  (i.e.,  equal-interval  or  ratio)  scale  scores  (Johnson 
&  Huber,  1977). 

The  Category  scaling  methods  represent  a  methodological  variant  of 
the  Ranking  methods.  In  these  methods,  alternatives  undergoing  evaluation 
are  sorted  into  discrete  categories.  The  categories  are  ordered  to  re¬ 
flect  a  continuum  of  worth  for  the  factor  under  consideration.  Category 
methods  are  often  easier  to  use  than  ranking  methods,  particularly  when 
evaluating  a  large  number  of  entities  (Torgerson,  1958).  They  are  also 
subject  to  the  same  limitations  as  the  Ranking  methods  in  that  the  result¬ 
ing  scale  values  are  ordinal  measures  of  relative  utility. 

Direct  scaling  methods  involve  the  assignment  of  numerical  scale 
values  to  the  levels  of  a  factor.  Typically,  a  decision-maker  is  asked 
to  assign  a  scale  value  to  each  factor  level;  the  scale  value  reflects  that 
level's  utility  relative  to  the  other  levels  of  the  factor.  In  MAUM,  the 
most  frequently  used  Direct  scaling  procedure  requires  a  decision-maker 
first  to  anchor  extreme  factor  levels  at  the  extremes  of  a  utility  scale, 
usually  a  O-to-100  range.  Intermediate  alternatives  are  then  scaled  be¬ 
tween  these  extreme  points  (Edwards,  1971). 

Although  the  Direct  scaling  methods  are  usually  used  to  assess  sub- 
jeative  utility,  they  also  can  be  used  to  assign  utility  scores  to  objective 
dimensions.  If  a  factor  can  be  scored  according  to  an  objective  criterion 
(e.g.,  cost),  the  resulting  scale  value  can  be  used,  along  with  utility 
scores  on  subjective  factors,  to  determine  aggregate  utility.  The  only 
requirement  is  that  all  of  the  values  be  made  commensurable  before  indi¬ 
vidual  partial  utilities  are  combined.  This  is  accomplished  by  convert¬ 
ing  all  scores  to  a  common  scale,  usually  O-to-100  (Edwards,  1971). 

The  Gamble  scaling  methods  involve  constructing  wagers  (defined  as 
a  situation  in  which  each  of  a  possible  set  of  outcomes  can  occur  with  a 
specified  probability)  and  then  vfrying  either  factor  levels  or  their  asso¬ 
ciated  probabilities  until  a  dccisicn-niaker  is  indifferent  between  the 
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wager  and  an  alternative  "sure  thing."  Gamble  methods  are  tlie  only  util¬ 
ity  assessment  procedure  having  a  basis  in  classical  utility  theory  (Keeney 
&  Raiffa,  1976).  They  are  also  the  only  methods  that  require  an  explicit 
assessment  of  the  uncertainty  associated  with  various  alternatives.  Hence, 
the  Gamble  methods  are  appropriate  for  MAUM  situations  involving  risk;  they 
are  not  applicable  in  riskless  situations. 

In  application,  Gamble  methods  have,  been  found  to  be  cumbersome  and 
time  consuming.  Raiffa  (1968)  has  referred  to  this  situation  as  the  "bushy 
mess"  problem:  in  many  practical  problems,  the  procedure  is  often  too  com¬ 
plex  to  carry  out.  Also,  in  many  applications,  subjective  estimates  of  the 
uncertainties  involved  (i.e.,  probabilities)  have  been  found  to  be  highly 
unreliable  (Keeney,  1977). 

Indifference  scaling  methods  represent  a  variation  on  the  Gamble 
methods.  In  these  procedures,  a  decision-maker  is  asked  to  determine  in¬ 
difference  points  between  combinations  and  levels  of  factors.  For  example, 
a  decision-maker  may  be  asked  to  indicate  whether  he  is  indifferent  between 
two  pairs  of  factor  level  combinations,  such  as  40%  reliability  and  a  repair 
cost  of  30  units  versus  50%  reliability  and  a  repair  cost  of  50  units.  In¬ 
difference  methods  are  the  only  scaling  procedures  applicable  in  situations 
involving  dependent  factors.  However,  methods  are  available  for  removing 
or  overcoming  factor  dependencies  [see  Dawes  &  Corrigan  (1974)  or  Einhorn  & 
Hogarth  (1975)].  Keeney  (1976,  p.7)  has  characterized  indifference  scaling 
methods  as  follows: 

"My  experience  with  (Indifference  scaling  methods)  suggests 
that  such  hypothetical  judgments  are  unreliable  and  un¬ 
representative  of  real  preferences;  worse,  they  bore  un¬ 
tutored  decision-makers  into  rejection  of  the  whole 
process  or  acceptance  of  answers  suggested  by  the  se¬ 
quence  of  questions  rather  than  answers  that  reflect 
their  real  values,  or  both." 

In  a  review  of  a  number  of  representative  MAUM  applications,  Johnson 
and  Huber  (1977)  conclude  that  Direct  scaling  methods  are  preferred  for  use 
in  MAUM  unless  the  decision  problem  involves  dependent  factors  that  cannot 
be  orthogonalized  or  involves  uncertainty  that  must  be  formally  considered. 
The  dominance  of  Direct  scaling  procedures  in  most  MAUM  situations  is  judged 
to  be  a  function  of  the  following  points: 

1.  Direct  scaling  m-'thods  are  simple,  easy  to  use,  have  high 
face  validity,  and  generate  quantitative  worth  scores. 

2.  Direct  scaling  methods  are  integral  to  many  frequently 
used  systems  of  worth  assessment  (for  example,  see 
Barlsh  and  Kaplan,  1978). 
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3.  The  results  of  utility  assessments  using  Direct  scaling 
methods  have  been  found  to  be  comparable  to  the  results 
of  assessments  made  using  more  sophisticated  procedures 
such  as  one  of  the  Gamble  methods  (Edwards,  1976). 

Table  1-1  presents  a  summary  overview  of  each  of  the  five  scaling  methods. 
The  table  lists  factors  that  should  be  considered  in  the  selection  of  a 
procedure  for  a  specific  MAUM  problem. 

Utility  Synthesis.  The  third  and  final  phase  of  the  MAUM  process  in¬ 
volves  combining  partial  utility  scores  to  produce  a  measure  of  aggregate 
or  global  utility  for  each  alternative.  Johnson  and  Huber  (1977)  present 
four  basic  models  for  combining  partial  utilities.  These  models  are  listed 
as  follows: 

1.  Linear  additive: 

n 

U  =  b  +  E  b,u(xj 

°  i=l  i  i 

2.  Linear  additive  with  interactions: 

n 

U  =  b^  +  I  b.u(x,)  +  EE  b,,u(x,)u(x,) 

°  i=l  ^  ^  ij^j  ^  J 

3.  Linear  additive  with  higher  order  functions: 

n 

U  =  b  +  E  b  f[u(x  )] 

i=l 

4.  Multiplicative: 

n  b 

U  =  b^  n  u(x  ) 

°  i=l  ^ 

In  the  above  equations,  U  represents  global  or  aggregate  utility,  u(x^) 
represents  the  partial  utility  of  |igving  attribute  i^  at  level  x^,  and  b^ 
represents  the  importance  of  the  i  factor  to  overall  utility. 
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Model  1,  linear  additive,  is  the  simplest  utility  ai',grey,at i on  model. 
Here,  the  u(x^)  are  the  simple  partial  utility  scores  for  each  of  the  fac¬ 
tors.  The  second  model,  linear  additive  with  interactions,  is  a  more  com¬ 
plex  version  of  model  1.  In  this  model,  interactions  between  various  factor 
combinations  are  also  considered.  Model  3,  linear  additive  with  hij^her  order 
functions,  permits  the  inclusion  of  complex  individual  factor  utility  func¬ 
tions  [e.g.,  u(x,)^,  logu(x.),  etc.].  This  model  is  also  applicable  to 
situations  in  which  individual  factors  are  multi-faceted;  that  is,  when 
Individual  partial  utility  scores  are  themselves  multiattribute  utility 
scores.  Model  4,  multiplicative,  represents  the  case  of  purely  interactive 
utility.  Multiplicative,  models  are  considerably  more  complicated  to  process 
than  the  additive  models.  Also,  models  of  this  kind  are  highly  sensitive 
to  changes  in  the  partial  utility  ratings  of  individual  factors. 

All  of  the  additive  models  (Models  1  through  3)  are  compensatory; 
that  is,  high  partial  utility  scores  on  one  or  more  factors  can  compensate 
for  low  ratings  on  other  factors.  The  multiplicative  model  is  far  less 
compensatory  than  the  additive  models  in  that  high  scores  on  one  or  more 
factors  do  not  as  readily  compensate  for  low  scores  on  other  factors. 

In  practical  applications,  additive  aggregation  models  have  been  found 
to  be  more  useful  than  more  complex  models  (Johnson  &  Huber,  1977).  This 
conclusion  is  based  on  the  fact  that  additive  models  are  simple,  quite  di¬ 
rect  in  their  application,  and  allow  decision-makers  a  perceived  high  de¬ 
gree  of  control  over  the  MAUM  process.  John  and  Edwards  (1978,  p.l)  pro¬ 
vide  the  following  comment  on  the  choice  of  a  simple  (i.e.,  not  involving 
interactions)  additive  model  in  MAUM;  "Most  choice  situations  are  non¬ 
additive,  risky,  or  both.  But  more  complicated  models,  involving  more 
complex  function  forms  and  more  parameters,  are  often  not  worth  the  effort. 

In  particular,  the  (simple)  additive  moael  serves  as  a  good  approximation 
to  much  more  complicated  forms  (Dawes,  19 '1;  Goldberg,  1965,  1968,  1970, 

1971;  Yntema  &  Torgerson,  1961)." 


Summary 


In  summary,  the  previous  discussion  made  a  case  for  the  use  of  informa¬ 
tion  utility  as  a  proxy  measure  for  strict  information  value  in  DORAC  eval¬ 
uations.  An  overview  of  utility  theory  and  the  technology  of  utility  meas¬ 
urement  was  also  presented.  These  results  lead  to  the  following  conclusions 
with  respect  to  the  use  of  MAUM  in  ClEA: 

1.  Problem  structuring  is  the  most  critical  aspect  of  utility 
measurement.  Thi.s  phase  of  the  analysis  is  also  the  least 
amenable  to  formal  prescription.  Currently,  no  universally 
applicable  procedures  for  utility  ,  oblem  structuring  exist. 


1-1? 


2.  In  terms  of  partial  utility  assessment,  Direct  scaling 
methods  have  been  found  to  be  most  useful.  The  Direct 
methods  are  flexible  In  that  they  are  applicable  to  ob¬ 
jective  as  well  as  subjective  situations.  Direct  scaling 
methods  also  result  in  quantitative  as  opposed  to  ordinal 
scale  scores. 

3.  Simple  additive  utility  aggregation  functions  have  been 
found  to  be  more  useful  than  more,  complex  combination 
functions.  One  simple  additive  aggregation  model  that 
holds  promise  for  use  in  CIEA  is  Model  3,  linear  additive 
with  higher  order  functions.  This  aggregation  model  will 
accommodate  a  direct  treatment  of  multidimensional  utility 
factors,  a  situation  almost  sure  to  be  encomitered  in 
DORAC  evaluations. 

The  next  section  of  the  report  presents  a  methodological  framework 
for  CIEA  that  incorporates  the  MAUM  procedures  described  in  this  section. 
The  CIEA  methodology  is  developed  within  the  context  of  a  traditional  cost- 
effectiveness  analysis  methodology  outlined  in  Kazanowski  (1968).  In  the 
CIEA  methodology,  the  effectiveness  measure — information  value. — is  derived 
through  a  MAUM  procedure  using  information  utility  as  a  proxy  measure  for 
strict  information  value.  To  illustrate  the  application  of  the  procedure, 
an  example  CIEA  is  presented  with  the  methodological  description. 
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SECTION  2 


COST  AND  INFORMATION  EFFECTIVENESS  ANALYSIS 
Introduction 


The  objective  of  a  Cost  and  Information  Effectiveness  Analysis  (CIEA) 
methodology  is  to  provide  the  framework  for  selecting  a  preferred  DORAC 
alternative  in  terms  of  the  value  of  the  information  provided  (e.g.,  opera¬ 
tional  readiness,  training  status,  etc.)  versus  the  coat  of  developing, 
implementing,  and  operating  the  capability.  CIEA  is  intended  to  serve  as 
a  guide  to  decision-makers  in  establishing  requirements  and  in  developing 
and  evaluating  alternative  DORAC  coucepta  for  any  parent  materiel  system. 

As  a  methodology,  CIEA  is  a  member  of  a  set  of  procedures  generally 
known  as  cost-effectiveness  analysis'.  The  term  cost-effectiveness  denotes 
a  technique  in  whicVi  alternative  systems  designed  to  accomplish  a  goal  are 
evaluated  using  measures  of  cost  (usually  dollars)  and  separate  measures 
of  effectiveness  (e.g.,  reliability,  speed,  probability  of  accomplishing 
a  task,  or  a  weighted  index  of  a  number  of  such  factors)  (Barish  &  Kaplan, 
1978).  Under  this  approach,  cost  and  effectiveness  values  for  each  alterna 
tive  are  determined.  The  systems  are  then  evaluated  on  the  basis  of 
whether  the  additional  benefits  of  the  more  effective  alternatives  are 
worth  their  added  costs.  The  use  of  cost-effectiveness  analysis  is  common 
in  the  evaluation  of  military  materiel  and  training  systems  (for  example, 
see  TRADOC  Pamphlet  11-8  or  TRADOC  Pamphlet  71-10) . 

The  CIEA  methodology  described  in  this  section  is  developed  within 
the  framework  of  a  general  cost-effectiveness  methodology  outlined  in 
Kazanowski  (1968).  The  phases  of  tha  methodology  are  listed  as  follows: 

1.  Assess  needs  and  constraints 

2.  Define  system  objectives 

3.  Identify  operational  requirements  for  the  attainment 
of  system  objectives 

4.  Develop  alternative  system  concepts 

5.  Establish  system  evaluation  criteria 

6.  Generate  systems-versus-criteria  array 

7.  Perform  sensitivity  analyses 

8.  Select  preferred  alternative 


1  ^ 
Technically,  CIEA  as  outlined  in  this  report  is  a  aost-benef%t 

analysis  because  a  number  of  effectiveness  measures  are  condensed  into 
a  single  composite  benefit  measure  that  serves  as  the  basis  for  the 
evaluation  of  alternatives. 
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In  line,  with  the  discussion  presented  in  Section  1,  a  MAUM  approach 
to  assessing  information  worth  is  integrated  into  the  CTEA  luethc iology . 

The  MAUM  process  used  in  CIEA  is  adapted  from  similar  procedures  outlined 
in  Edwards  (1976)  and  Johnson  and  Huber  (1977).  The  corresponding  phases 
in  the  M.\UM  process  are  listed  as  follows; 

1.  Detei'mine  the  issues  to  which  utili*”  ratings  are  relevant. 

2.  Determine  the  relevant  factors  on  which  alternative  system 
concepts  are  to  be  assessed. 

3.  Identify  the  perspective  from  which  utility  is  to  be  assessed. 

4.  Develop  operational  measures  for  each  factor. 

5.  Identify  the  alternatives  to  be  evaluated. 

6.  Derive  importance  weights  for  each  factor. 

7.  Assess  utility  of  each  operational  measure. 

8.  Obtain  system  effectiveness  ratings. 

9.  Determine,  partial  utility  scores  for  each  factor. 

10.  Aggregate  partial  utilities  to  produce  global  utility 
score  for  each  alternative. 

11.  Estimate  cost  for  each  alternative. 

12.  Select  a  preferred  alternative. 

Figure  2-1  presents  the  phases  in  the  MAUM  procedure  keyed  to  the  phases 
in  CIEA  (denoted  MAUM-CIEA) . 

In  this  report,  the  emphasis  of  the  methodological  discussion  is  on 
the  conduct  of  CIEA  for  fielded  materiel/ training  device  systems.  Pro¬ 
cedures  for  the  conduct  of  CIEA  on  unfielded  systems  represent  a  subset 
of  the  complete  process,  but  are.  not  addressed  explicitly  in  this  report. 
The  complete  CIEA  procedure  is  presented  and  discussed  in  the  remainder 
of  this  section, 


Analytic  Methodology 


The  following  subsection  presents  a  discussion  of  the  activities  in¬ 
volved  in  the  conduct  of  a  CIEA.  To  aid  in  the  explication  of  the  method¬ 
ology,  an  exemplary  analysis  (the  boxed-in  sections)  is  presented  with  the. 
narrative  description  for  each  phase.  Each  of  the  distinct  phases  of  CIEA 
is  now  discussed  in  turn. 


Assess  .deeds  and  Constraints 


Phase  one  of  the  CIEA  process  Involves  establishing  the  need  for  a 
DORAC  and  defining  general  constraints  that  will  serve  as  a  guide  for  the 
analysis.  The  impetus  for  DORAC  development  will  generally  come  from  a 
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leadership  concern  that  readiness  evaluation  or  reporting  for  a  particular 
materiel  system  (e.g.,  M-16  Rifle,  Infantry  Righting  Vehicle,  etc.)  is  not 
being  done  adequately.  Leadership  concern  may  arise  from  a  number  of 
sources.  For  example,  it  may  result  from  the  fact  that  Unit  Status  Reports 
(USRs)  do  not  agree  with  Skill  Qualification  Test  (SQT)  results.  Army  Train¬ 
ing  and  Evaluation  Program  (ARTEP)  reports,  or  results  from  other  individual 
or  unit  exercises.  In  otlier  situations,  the  judgment  that  "tilings  are  not 
right"  may  be  based  primarily  on  commanders'  subjective  opinions.  In  yet 
other  cases,  ammunition  and  other  costs  and  constraints  may  limit  the  fre¬ 
quency  with  which  information  on  current  status  can  be  obtained.  Whatever 
the  source  or  basis,  the  impetus  for  the  development  of  a  D0R.4C  will  be 
generated  by  the  identification  or  perception  of  a  possible  readiness  de¬ 
ficiency  that  is  judged  to  have  significant  impact  on  the  Army's  fighting 
ability . 

After  the  need  for  a  DORAC  is  established,  the  next  aspect  of  needs/ 
constraints  assessment  is  a  formal  recognition  of  need  in  the  form  of  a 
problem  statement.  The  problem  statement  should  define,  in  general  terms, 
the  basis  for  leadership  concern  and  address  possible  sources  of  the  per¬ 
ceived  inadequacies  in  the  readiness  evaluation  or  reporting  systems  (e.g., 
no  data  are  collected,  the  wrong  data  are  collected,  the  data  that  are 
collected  are  invalid  or  unreliable,  data  are  not  collected  often  enough, 
etc. ) . 

CIEA  must  address  a  real  world  situation  in  which  constraints  exist. 
Thus,  after  recognizing  the  need  for  improved  readiness  reporting,  a  third 
aspect  of  needs/constraints  assessment  is  to  identify  potential  constraints 
on  the  development  or  deployment  of  such  a  DORAC.  The  principal  categories 
of  constraints  that  should  be  considered  include  the  following  (TRADOC 
Pamphlet  11-8)  : 

1.  Economic,  Tlie  DORAC  must  be  developed  and  implemented 

within  the  context  of  the  economics  that  apply  to  the 
time  period  being  considered. 

2.  Military.  Complex  alternatives,  no  matter  how  attractive, 

will  usually  have  to  be  operated  and  maintained  by 
Army  personnel. 

3.  Technological.  The  DORAC  sliould  be.  constrained  by  the  use 

of  demonstrated  technology.  Alternatives  that  push 
the  technical  state  of  the  art  should  be  given  careful 
review.  Unattainable  technological  objectives  can 
result  in  the  loss  of  time,  money,  and  effort. 

4.  Personnel,  Personnel  resources  are  scarce  even  in  wartime. 

Evaluation  systems  that  require  significantly  more 
personnel  than  those  they  replace  should  be  considered 
carefully. 
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5.  Time.  All  changes  require  time  to  Implement,  The  CIEA 

should  consider  tlie  time  frame  for  the  development  and 
deployment  of  a  DORAC.  These  times  should  be  evaluated 
with  an  eye  toward  the  developmental  stage  of  the 
parent  materiel  system.  For  example,  if  a  materiel 
system  is  scheduled  for  replacement  beginning  in 
fiscal  year  (FY)  1983,  it  makes  little  sense  to 
develop  a  DORAC  that  will  not  be  fully  deployed  until 
FY  1982,  unless  the  capability  can  also  be  used  with 
the  system's  replacement. 

It  is  doubtful  that  all  relevant  constraints  can  be  identified  during 
the  needs/constraints  phase  of  the  analysis.  However,  applicable  categories 
of  constraints  should  be  specified.  One  constraint  that  should  be  con¬ 
sidered  during  needs/constraints  assessment  is  system  cost.  Cost  will 
constrain  the  kinds  of  DORACs  that  are  deployed.  Hence,  benchmark  cost 
guidelines  should  be  provided  early  in  the  analysis.  Early  determination 
of  cost  constraints  will  serve  to  eliminate  excessively  costly  alternatives 
early  in  their  developmental  cycle. 


Define  System  Objectives 

The  second  phase  in  CIEA  is  to  refine  the  problem  statement  developed 
in  the  previous  phase  into  a  formal  specification  of  the  objectives  of  the 
proposed  DORAC.  Phase  2  is  carried  out  in  three  separate  steps,  listed  as 
follows : 

1.  Determine  the  issues  to  which  DORAC  information  is  relevant, 

2.  Specify  information  worth  dimensions. 

3.  Identify  the  perspective  from  which  information  worth 
is  to  be  assessed. 

Step  one  involves  answering  the  question:  "What  purposes  are  the 
DORAC  data  to  serve?"  This  question  is  addressed  by  developing  a  list  of 
information  issues  (e.g.,  can  our  riflemen  be  used  effectively  in  assault 
operations?  Can  our  riflemen  be  expected  to  engage  hostile  targets  pre¬ 
senting  the  greatest  threat?)  from  the  problem  statement.  The  information 
issues  are  similar  in  purpose  to  the  essential  elements  of  analysis  in  Cost 
and  Training  Effectiveness  Analysis  (CTEA) ;  they  serve  to  guide  the  defini¬ 
tion  of  criteria  for  information  worth  evaluation. 

After  a  list  of  information  issues  has  been  developed,  the  second  step 
of  phase  2  is  to  organize  the  information  issues  into  a  set  of  worth  dimen- 
v.lons  (WDs),  or  categories  of  Information  use.  The  WDs  constitute  the  pri¬ 
mary  value  factors  for  the  evaluation  of  DORAC  alternatives.  Examples  of 
some  potential  DORAC  worth  dimensions  are  provided  as  follows: 
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1.  Readiness  Evaluation.  The  determination  of  v^hether  or  not 
individuals/units  are  capable  of  performance  at  an  accept¬ 
able  level/standard  on  performances  specific  to  the  DORAC 
information  issues. 

2.  Training  Management.  The  use  of  training  status  and  per¬ 
formance  diagnostic  information  in  determining  who,  how 
often,  when,  and  what  to  train  for  individual/unit 
performances  related  to  the  specific  DORAC. 

3.  Unit  Management.  The  use  of  objective  job  performance  in¬ 
formation  to  provide  guidance  in  various  unit  management 
activities  such  as  the  award  of  performance  incentives,  the 
assignment  of  personnel  to  critical  unit  positions,  and 

so  forth. 

4.  Fighting  System  Evaluation/Development.  The  use  of 
evaluation  data  to  provide  feedback  to  branch  schools 
and  other  concerned  agencies  on  training  program  content, 
training  materials,  training  devices,  system  equipment, 
support  equipment,  doctrine,  tactics,  and  so  forth. 

Progressing  from  information  issues  to  WDs  is  likely  to  lead  to  the 
Identification  of  additional  information  issues.  Hence,  phases  one  and 
two  should  be  performed  iteratively  until  all  pertinent  information  issues 
and  WDs  are  identified. 

In  developing  WDs  it  is  necessary  not  to  be  too  expansive.  For  reasons 
that  will  be  made  apparent  later,  the  number  of  WDs  should  not  exceed  seven. 
If  more  than  seven  WDs  are  developed,  the  list  should  be  reviewed  and  the 
number  of  WDs  reduced  by  redefining  and  combining  dimensions.  For  most 
applications,  the  WDs  listed  above  should  suffice. 

The  last  step  in  phase  2  involves  identifying  the  perspective  from 
which  information  utility  is  to  be  assessed.  Considering  the  WDs  developed 
in  step  2,  organizations  having  a  stake  in  determining  the  value  of  DORAC 
information  are  specified.  When  several  organizational  perspectives  are 
indicated,  persons  who  can  speak  for  the  various  stakeholders  must  be 
identified  and  Induced  to  participate  in  the  evaluation. 


Identify  Operational  Requirements  for  the  Attainment  of  System  Objectives 

Operational  requirements  are  derivatives  of  system  objectives  that 
specify  precisely  what  measurement  capability  is  required  on  the  part  of 
a  DORAC.  The  objective  of  phase  3  is  to  translate  the  system  objectives 
(i.e.,  information  issues)  developed  in  phase  2  into  specific  operational 
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requirements  for  the  DORAC.  This  includes  specifying:  (1)  What  per forir 
ances  are  to  be  evaluated,  (2)  the  conditions  under  which  the  performances 
must  be  demonstrated,  and  (3)  the  standards  by  which  performances  are  eval¬ 
uated.  Each  of  these  aspects  of  the  operational  requirements  specification 
is  now  discussed  separately. 

Performances .  The  first  step  in  phase  3  is  to  state  precisely  whc  ' 
performances  are  implied  in  the  information  issues.  These  should  be  stated 
as  a  list  of  relevant  performance  sta.ements.  Examples  of  acceptable  per¬ 
formance  statements  are  listed  as  follows: 

Climb  the  telephone  pole 
Disassemble  an  M-16A1  rifle 
Camouflage  the  helmet 
Add  two  five-digit  numbers 

Note  that  each  performance  statement  includes  an  action  verb.  The  action 
verb  is  the  key  to  the  performance;  it  tells  exactly  what  must  be  done. 

For  example,  in  the  statement  "Disassemble  an  M-16  A1  rifle,"  the.  action  verb 
is  "disassemble".  It  is  possible  to  assess  directly  an  individual's 
ability  to  perform  this  action. 

Conditions .  Each  performance  statement  must  include  a  statement  of 
the  conditions  under  which  the  performance  is  to  be  demonstrated.  The  con¬ 
dition  statement  should  indicate  the  following  aspects  of  each  performance : 

1.  What  an  evaluates  has  to  work  with  (e.g.,  tools,  reference 
materials,  etc.). 

2.  The  environmental  circumstances  under  which  the  performance 
must  be  demonstrated  (e.g.,  nighttime,  daylight,  rain,  etc.). 

3.  What  the  evaluates  must  work  on — his  starting  point  (e.g., 

"given  an  M-16A1  rifle") . 

4.  Any  limitations  or  special  considerations. 

It  is  important  to  specify  all  conditions  that  are  relevant  to  per¬ 
formance  assessment  using  the  DORAC.  In  many  cases,  performances  may  be 
demonstrated  under  a  number  of  conditions.  The  statement  of  conditions 
pertinent  to  DORAC  evaluation  should  make  clear  which  conditions  are  to  be 
considered . 

Standards .  Finally,  each  performance  statement  must  specify  the 
standard  or  criterion  by  which  the  performance  is  evaluated.  The  standard 
indicates  how  Welt  or  haw  quickly  (or  both)  a  performance  is  to  be  done. 

Six  categories  of  standards  are  usually  employed  to  indicate  how  well  or 
how  quickly  a  performance  must  be  done.  These  categories  of  standards 
are  listed  as  follows  (Swezey  &  Pearlstein,  1974): 
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1.  Standard  Operating  Procedure  (SOP) — performance  must  match 
a  specified  SOP.  The  standard  specifies  when  a  performance 
is  complete  and  its  proper  sequence. 

2.  Zero  error — performance  must  be  completed  with  100%  accuracy. 

•3.  Minimum  acceptable  level — performance  must  meet  a  specified 
minimum  acceptable  level. 

4.  Subjective  quality — performance  must  match  stated  qualitative 
characteristics . 

5.  Time  requirements — performance  must  be  done  within  a 
given  time  period. 

6.  Production  rate — performance  must  yield  a  specified  output 
per  unit  of  time. 

In  most  situations  involving  fielded  materiel  systems,  existing  task 
analysis  documentation  will  provide  the  information  necessary  to  develop 
performance  statements,  conditions,  and  standards  (i.e.,  performance  ob¬ 
jectives).  Situations  may  be  encountered,  however,  (e.g.,  when  working 
with  an  unfielded  materiel  system)  in  which  performance  objectives  will 
have  to  be  developed  by  the  analyst.  As  in  CTEA,  performances,  conditions, 
and  standards  for  unfielded  systems  can  be  developed:  (1)  from  a  knowledge 
of  antecedent  systei.'.c,  (2)  on  the  basis  of  available  system  development 
documentation  [e.g.,  LSAR  (Logistics  Support  Analysis  Record)  documentation], 
or  (3)  from  judgments  rendered  by  subject  matter  experts. 

As  noted  previously,  performance  objectives  (i.e.,  tasks,  conditions, 
and  standards)  are  derivatives  of  the  inforr  ation  issues  developed  in 
phase  2.  The  performance  objectives  const^  “e  an  operational  definition 
of  precisely  what  is  implied  in  each  of  the  ii.  ormation  issues.  As  such, 
the  performance  objectives  represent  the  operational  measures  upon  which 
the  evaluation  of  each  DORAC  alternative  is  based.  The  application  of  each 
component  of  the  performance  objectives  are  addressed  later  in  the  re  'rt 
at  appropriate  places  in  the  discussion.  For  clarification  purposes. 

Figure  2-2  depicts  the  relationship  between  the  problem  statement,  WDs, 
information  issues,  and  performance  statements,  conditions,  and  standards. 


Develop  Alternative  System  Concepts 

Phase  4  is  concerned  with  characterizing  alternative  DORAC  concepts 
'^or  meeting  the  operational  requirements  developed  in  Step  3.  In  this 
step,  information  concerning:  (1)  what  is  required  to  be  measured  (i.e.. 
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Figure  2-2.  The  Relationship  Between  the  Problem  Statement,  Worth  Dimensions, 
Information  Issues,  and  Performance  Objectives 


performance  statements),  (2)  the  conditions  under  which  performances  are 
to  be  done,  and  (3)  performance  standards  are  used  to  specify  operational 
measurement  capabilities.  Differences  in  operational  measurement  capa¬ 
bilities  define  DORAC  alternatives. 

In  considering  the  operational  characteristics  of  DORAC  alternatives, 
the  first  differentiating  variable  is  performance  assessment  capability; 
that  is,  whether  or  not  an  alternative  provides  a  vehicle  for  assessing  a 
particular  performance.  To  document  assessment  capabilities,  DORAC  al¬ 
ternatives  are  scored  using  a  Performance  by  Alternative  matrix.  Entries 
in  this  matrix  are  either  "X"  or  "0",  indicating  that  an  alternative  either 
does  or  does  not  provide  a  vehicle  for  assessing  a  given  performance. 

Figure  2-3  presents  a  hypothetical  Performance  by  Alternative  matrix. 

The  second  differentiating  variable  for  DORAC  alternatives  involves 
performance  conditions.  Working  from  the  performances  that  can  be 
assessed  using  one  or  more  of  the  DORAC  alternatives,  the  relevant  condi¬ 
tions  for  each  are  listed.  Each  DORAC  alternative  is  then  further  charac¬ 
terized  according  to  whether  or  not  it  provides  a  vehicle  for  assessing  a 
given  performance  under  each  relevant  condition.  Again,  an  "X"  or  "0" 
s''orlng  scheme  is  used  to  characterize  each  alternative.  The  conditions 
ta  are  recorded  in  the  Performance  by  Condition  matrix.  A  separate 
rformance  by  Condition  matrix  is  produced  for  each  DORAC  alternative. 

)  '’ure  2-4  presents  an  exemplary  Performance  by  Condition  matrix. 

The  previous  incidence  matrices  (Figures  2-3  and  2-4)  characterize 
DORAC.  alternatives  according  to  their  assessment  potential;  that  is,  by 
whet  er  or  not  it  is  possible  to  assess  specific  performances  under  given 
conditions  using  a  particular  alternative.  Along  with  the  incidence  data, 
it  also  necessary  to  specify  hou  performance  assessment  is  done.  Ac¬ 
cordingly,  each  cell  in  the  Performance  by  Alternative  matrix  containing 
an  "X"  (i.e.,  assessment  ^  possible)  is  next  characterized  according  to 
how  performance  assessment  is  accomplished.  For  example,  potential  methods 
of  performance  measurement  Include  the  following: 

1.  Observer  verification, 

2.  Performance  checklist, 

3.  Numerical  rating  scale, 

4.  Machine  recording  with  observer  summary  report, 

5.  Machine  recording  with  hardcopy  trace  and  summary  report. 

Each  assessment  method  is  rated  according  to  the  judged  dependability 
of  the  data  it  will  provide.  Dependability  ratings  are  made  using  the 
following  5-point  scale,  presented  as  follows: 
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1.  Very  low 

2 .  Low 

3.  Moderate 
A.  High 

5.  Very  high 

Factors  that  should  be  considered  in  assigning  dependability  ratings  in¬ 
clude  : 

1.  The  judged  validity  of  the  resulting  data.  Is  the  recorded 
score  likely  to  be  a  valid  characterization  of  true 
evaluatee  performance? 

2.  The  inherent  stability  (i.e.,  reliability)  of  the  method. 

3.  The  judged  precision  of  the  method.  Does  the  assessment 
methodology  provide  data  with  sufficient  precision  to 
establish  whether  or  not  the  performance  standard  is  met? 

A.  The  amount  of  human  intervention  in  the  measurement  process. 

Data  dependability  is  directly  proportional  to  the  first  three  factors  and 
inversely  related  to  the  third. 

The  last  step  in  Phase  A  involves  developing  an  evaluation  scenario 
for  each  DORAC  alternative.  This  scenario  outlines  characteristics  of  the 
evaluation  situation  such  as;  (1)  whether  or  not  all  performances  are  to 
be  assessed  during  each  evaluation  (this  will  impact  upon  ^.he  time  required 
to  conduct  an  evaluation),  (2)  the  number  of  planned  evaluations  per  year, 
(3)  the  anticipated  number  and  type  of  evaluators  per  evaluation  position, 
(A)  whether  or  not  the  collection  of  evaluation  data  is  part  of  regularly 
scheduled  training,  and  so  forth.  Wlien  combined  with  the  operational 
characteristics  of  each  proposed  alternative,  the  evaluation  scenarios 
provide  the  basis  for  generating  cost  estimates  for  DORAC  alternatives. 

The  position  requirements  portion  (i.e.,  steps  1  through  3)  of  the  DORAC 
Costing  Guidelines  presented  in  Appendix  A  should  prove  useful  in  develop¬ 
ing  evaluation  scenarios. 


At  this  point  in  the  discussion,  an  illustration  CIEA  is 
introduced.  The  example  exercise  is  structured  around  the  con¬ 
duct  of  an  analysis  for  a  hypothetical  set  of  training  device 
alternatives  for  the  M-16A1  rifle  plus  the  baseline  field 
"record  fire"  procedures.  It  should  be  noted  that  a  baseline 
case  consisting  of  the  current  evaluation  procedure  will  often 
be  used  in  CIEA.  For  purposes  of  analysis,  a  needs/constraints 
assessment  (phase  1)  is  assumed;  it  is  further  assumed  that  in¬ 
formation  issues  have  been  specified  and  have  led  to  the 
development  of  two  WDs,  listed  as  follows: 
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1.  Readiness  Evaluation 

2.  Training  -anagement 

An  analysis  of  UORAC  objectives  has  resulted  in  the  follow¬ 
ing  performance  statements: 

1.  Load/unload  rifle  magazine 

2.  Reduce  stoppage  and  clear  rifle 

3.  Zero  rifle 

A.  Engage  stationary  targets  under  field-fire  (FF) 
conditions 

5.  Engage  moving  targets  under  FF  conditions 

6.  Engage  moving  targets  from  defensive  position 

7.  Engage  moving  targets  in  assault  mode 

The  performance  statements  categorized  by  WDs  are  listed  as 
follows : 

Readiness  Evaluation 

1.  Engage  stationary  targets  under  FF  conditions. 

2.  Engage  moving  targets  under  FF  conditions. 

3.  Engage  moving  targets  from  defensive  position. 

A.  Engage  moving  targets  in  asvsault  mode. 

Training  Management 

1.  Load/unload  rifle  magazine. 

2.  Reduce  stoppage  and  clear  rifle. 

3.  Zero  rifle. 

A.  Engage  stationary  targets  under  FF  conditions. 

5.  Engage  moving  targets  under  FF  conditions. 

Performance  conditions  and  standards,  other  than  those  implied 
in  the  performance  statement,  are  not  explicitly  considered  in 
the  example  exercise. 

Phase  A,  Develop  Alternative  System  Concepts,  is  used  to 
characterize  three  hypothetical  alternatives,  denoted  B  (base¬ 
line),  X,  and  Y.  Figure  2-5  presents  the  Performance  by 
Alternative  matrix  for  the  three  hypothetical  alternatives. 
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Alternative 
B  X 


Y 


1. 

Load/unload 

X 

X 

X 

2. 

Reduce  stoppage 

X 

X 

X 

3. 

Zero 

X 

X 

X 

Performance  A. 

Stationary — FF 

X 

X 

X 

5. 

Moving-FF 

0 

0 

X 

6. 

Moving-defense 

0 

0 

X 

7. 

Moving-assault 

0 

0 

X 

Figure  2-5.  Performance  by  Alternative  Matrix 
for  CIEA  FiXample  Exercise 


Measurement  method  and  data  dependability  evaluations  for  the 
three  alternatives  are  presented  in  Figure.  2-6, 

Alternative 


B 

X 

Y 

1. 

Load /unload 

Observer 
verif ica- 
tion(OV)/ 
Moderate 
(Mod) 

OV/ 

Mod 

OV/ 

Mod 

2. 

Reduce  stoppage 

CV/ 

Mod 

OV/ 

Mod 

OV/ 

Mod 

3. 

Zero 

OV/ 

Mod 

Machine 

scored/ 

High 

Machine 

scored/ 

High 

Performance  4. 

Stationary — FF 

OV/ 

Mod 

Machine 

scored/ 

Mod 

Machine 

scored/ 

High 

5. 

Moving — FF 

Machine 

scored/ 

High 

6. 

Moving-defense 

Machine 

scored/ 

High 

7. 

Moving-assault 

Machine 

scored/ 

High 

Figure  2-6.  Measurement  Method/Data  Dependability 
Evaluations  for  CIEA  Example  Exercise 
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The  final  product  of  phase  4  is  the  evaluation  scenario.  It  was 
specified  that  each  performance  be  evaluated  during  eaeli  evalua¬ 
tion  period;  furthermore,  each  alternative  is  to  be  employed  with 
the  following  frequency; 


Alternative 

Frequency 

1. 

B  (Baseline) 

Once  per  year 

2. 

X 

Twice  per  year 

3. 

Y 

Quarterly  (four  times  per  year) 

Establish  System  Evaluation  Criteria 

The  next  phase  in  CIEA  involves  rating  the  system  evaluation  criteria- 
WDs  and  OMs — in  terms  of  their  importance  and  utility,  respectively.  First 
the  WDs  are  rated  according  to  their  importance  to  overall  DORAC  value. 

In  step  two,  the  OMs  nested  within  WDo,  are  rated  according  to  their  util¬ 
ity  for  decision-making.  Tlie  rating  techniques  employed  are  adapted  from 
similar  procedures  presented  in  Churchman,  Ackotf,  and  Arnoff  (1957), 
Edwards  (1976),  and  Barish  and  Kaplan  (1978). 

Worth  Dimensions.  The  first  step  in  establishing  system  evaluation 
criteria  involves  determining  the  Importance  of  each  of  the  WDs  to  over¬ 
all  DORAC  value.  Importance  weights  for  the  WDs  are  obtained  using  the 
series  of  substeps  presented  below: 

1.  Rank  the  WDs  in  order  of  importance.  The  ranking 

can  be  performed  either  by  an  individual,  or  by  repre¬ 
sentatives  of  various  stakeholders  acting  as  a  group. 

The  group  process  can  be  formal,  as  in  Delphi  or  the 
Nominal  Group  Technique,  or  informal.  If  a  group  pro¬ 
cedure  is  employed,  the  final  rankings  should  reflect 
a  consensus  of  opinion, 

2.  Rate  the  WDs  on  importance,  preserving  ratios. 

a.  Assign  the  least  Important  WD  a  rating  of  10. 

b.  Consider  the  next-least-important  WD.  How  much 
more  important  is  it  than  the  least  j.mportant? 

Assign  it  a  number  that  reflects  that  ratio.  For 
example,  if  the  sccond-least-important  W’D  is  judged 
to  be  four  times  as  Important  as  the  first,  it  is 
assigned  a  score  of  40.  Continue  up  the  list  of 
WDs,  checking  each  set  of  ratios  as  each  new  judgment 
is  made. 
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c.  Sum  the  resulting  importance  scores,  divide  each  by 
the  sum,  and  multiply  by  100. 

Reevaluate  importance  ratings  to  improve  reliability. 

a.  Compare  the  ^irst  (most  important)  WD  with  the  remain¬ 
ing  ones  pur,  together.  Is  it  more  important,  equally 
important,  or  less  Important  than  all  the  others  put 
together? 

b.  If  the  first  WD  is  more  important  than  all  of  the 
others  put  together,  see  if  it's  importance  rating 

is  greater  than  the  sum  of  the  importance  ratings  for 
all  of  the  ether  WDs.  If  not,  change  the  importance 
rating  of  the  first  WD  so  that  it  is  greater  than  the 
sum  of  the  others. 

c.  If  the  first  WD  is  of  equal  importance  to  all  the 
others  put  together,  see  if  its  importance  rating 

is  equal  to  the  sum  of  the  importance  ratings  of  all 
the  other  WDs.  If  it  is  not,  change  the  importance 
rating  of  the  first  WD  so  that  it  is  equal  to  the  sum 
of  the  others. 

d.  If  the  first  WD  is  less  important  than  all  the  others 
put  together,  see  if  its  importance  rating  is  less 
than  the  sura  of  the  importance  ratings  of  all  of  the 
other  WDs.  If  it  is  not,  change  the  importance  rating 
of  the  first  WD  so  that  it  is  less  than  the  sum  of 
the  others. 

e.  If  the  first  WD  was  considered  more  important  or 
equally  important  than  all  the  others  put  together, 
apply  the  above  procedure  to  the  second-most-important 
WD  on  the  list.  Is  it  more  important,  equally  im¬ 
portant,  or  less  important  than  all  the  other  farther 
down  the  list  put  together?  Then,  proceed  as  in  (b) , 
(c) ,  and  (d)  above,  applying  the  revision  procedure 

to  the  second  WD  instead  of  the  first. 

f.  If  the  first  was  considered  less  important  than 
all  the  others  put  together,  compare  the  first  WD 
with  all  the  remaining  ones  put  together,  except  the 
lowest  rated  one.  Is  the  first  WD  more  important, 
equally  important,  or  less  important  than  all  of  the 
others  farther  down  the  list  except  the  lowest  one 
put  together?  Then  proceed  as  in  (b) ,  (c) ,  and  (d) 
above.  If  (b)  or  (c)  are  applicable,  proceed  to  (e) 
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after  applying  (b)  or  (c).  If  (d)  is  applicable, 
proceed  as  in  this  paragraph  (f)  again,  comparing 
the  first  WD  with  all  the  remaining  ones  put  together 
except  the  lowest  two.  As  long  as  (d)  is  applicable, 
the  procedures  of  this  paragraph  (f)  are  repeated 
until  the  first  WD  is  compared  with  the  second  and 
third  ITOs  put  together.  Then,  even  if  (d)  is  still 
applicable,  proceed  to  (e). 

g.  Continue  the-  above  procedure  until  the  third-f rom-the- 
lowest  WD  has  been  compared  with  the  two  lowest  WDs 
on  the  list. 

4.  Repeat  action  (2-c)  on  the  revised  importance  ratings  pro¬ 
duced  in  substep  (3)  except  do  not  multiply  by  100.  The 
results  of  this  substep  (4)  are  the  importance  weights  for 
each  WD. 

A  sample  Rating  Development  Sheet  (RDS)  is  shown  as  Figure  2-7.  On  the 
RDS,  space  is  provided  for  no  more  than  seven  WDs.  When  more  than  seven  WD 
are  included,  the  rating  procedure  becomes  cumbersome.  If  more  than  seven 
initial  ITOs  are  developed,  the  decision-making  individual  or  group  should 
review  the  list  of  WDs  and  reduce  their  number  to  seven  or  fewer  by  elim¬ 
inating  the  least  important  dimensions  or  by  combining  dimensions.  If  it 
is  not  possible  to  reduce  the  list  of  WDs  to  seven  or  fewer,  a  final  course 
of  action  is  to  omit  substep  (3),  and  retain  the  initial  ratings  from  sub¬ 
step  (2).  However,  stopping  with  substep  (2)  can  result  in  importance 
weights  that  are  less  reliable  than  weights  obtained  through  the  exercise 
of  the  entire  procedure. 

Operational  Measures.  The  second  step  in  establishing  system  eval¬ 
uation  criteria  involves  determining  the  utility,  or  value  foy  deaision- 
making ,  of  the  OMs  nested  within  each  of  the  WDs.  Utility  scores  for  OMs 
are  determined  using  the  following  series  of  substeps: 

1.  List  the  OMs  in  descending  order  of  utility  (i.e.,  value 
for  decision  making)  to  the  WD  being  considered. 

2.  If  there  are  seven  or  fewer  OMs  nested  within  a  WD,  obtain 
utility  scores  for  each  following  the  procedure  outlined 
in  the  previous  section  (i.e.,  for  weighting  WDs). 

3.  If  a  WD  is  characterized  by  more  than  seven  OMs,  obtain 
utility  scores  as  follows: 

a.  Select  one  OM  at  random. 


2-18 


Figure  2-7.  Sample  Rating  Development  Sheet 


b.  Randomly  assign  each  of  the  remaining  OMs  to  groups 
of  approximately  equal  size,  with  no  more  than  five 
OMs  to  a  group. 

c.  Add  the  selected  OM  (a)  to  each  group  and  assign  it 
a  rating  of  100.  This  OM  will  serve  to  link  each  of 
the  groups  for  a  later  recombination  of  the  ratings 
(action  3-f). 

d.  Rank  each  of  the  OMs  in  each  group  in  order  of  descend¬ 
ing  utility.  Then,  assign  ratings  to  them  following 
substeps  2  and  3  of  the  WD  weighting  procedure  outlined 
previously.  In  each  group,  keep  the  rating  of  the 
selected  OM  (a)  at  100. 

e.  For  each  group,  follow  the  procedure  described  in 
substep  3  of  the  WD  weighting  procedure.  Do  not  change 
the  rating  of  100  assigned  to  the  selected  OM  (a)  and 
do  not  adjust  the  ratings  (substep  4). 

f.  Make,  a  combined  listing  of  all  the  OMs  in  order  of 
decreasing  utility.  Compare  this  listing  with  the 
initial  rankings  from  substep  1.  Note  any  difference 

in  rankings.  If  the  initial  list  (1)  is  judged  correct, 
repeat  actions  (a)  through  (e)  to  adjust  the  affected 
groups  and  reconcile  the  evaluations. 

g.  Adjust  the  utility  ratings  as  described  previously 
(action  2-c,  p.  2-17).  Do  not  multiply  the  results 
by  100. 

4.  Repeat  substeps  1  through  3  for  each  WD. 

The  results  of  CIEA  phase  5  are  a  set  of  importance  weights  for  WDs 
and  a  set  of  utility  scores  for  OMs  nested  within  WDs.  These  values  are 
used  later  in  the  analysis  to  produce  partial  utility  scores  for  each  WD, 
and  to  combine  partial  utility  scores  to  produce  global  utility  scores  for 
DORAC  alternatives. 


To  illustrate  phase  5,  the  example  CIEA  is  continued  in  the 
following  paragraphs.  The  first  step  Involves  rating  the  WDs 
acc>'rding  to  their  importance  to  overall  DORAC  value.  Accordingly, 
Readiness  Evaluation  (RE)  and  Training  Management  (TM)  are  ranked 
one  and  two,  respectively.  Furthermore,  RE  is  judged  to  be  four 
times  as  important  as  TM.  The  results  of  step  one  are  summarized 
as  follows : 
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Initial 

Scaled 

Worth  Dimension 

Rank 

Importance 

Rating 

Importance 

Rating 

Readiness  Evaluation 

1 

40 

.80 

Training  Management 

2 

10 

.20 

Step  two  of  phase  5  is  concerned  with  establishing  the  utility 
for  decision-making  of  each  OM  nested  within  each  WD.  This  step 
is  conducted  in  two  substeps.  The  first  substep  is  to  obtain  a 
set  of  initial  scaled  utility  ratings  by  applying  the  rank-rate 
procedure  outlined  above.  Results  from  substep  one  are  presented 
as  follows: 


Worth  Dimension  Rank 

Readiness  Evaluation 


1. 

Moving-assault 

1 

2. 

Moving-defense 

2 

3. 

Moving-FF 

3 

4. 

Stationary-FF 

4 

Training  Management 

1. 

Moving-FF 

1 

2. 

Stationary-FF 

2 

3. 

Zero  rifle 

3 

4. 

Reduce  stoppage 

4 

5. 

Load/unload 

5 

Initial 

Utility  Initial  Scaled 

Rating  Utility  Rating 


100 

43 

75 

32 

50 

21 

10 

4 

200 

54 

100 

27 

50 

14 

10 

3 

10 

3 

Following  the  development  of  an  initial  set  of  scaled  utility 
ratings  for  the  OMs ,  substep  two  of  step  two  is  initiated.  The 
objective  of  substep  two  is  to  refine  the  output  of  substep  one 
into  utility  scores.  Considering,  for  example,  the  initial  scaled 
utility  ratings  for  the  OMs  nested  under  Readiness  Evaluation  (see 
the  table  below),  moving-assault  is  judged  equal  in  utility  to  the 
remainder  of  the  OMs.  Thus,  the  utility  rating  for  moving  assault 
is  changed  from  43  to  57,  the  sum  of  the  ratings  for  the  other  OMs. 
Proceeding  do\m  the  list,  moving-defense  is  judged  to  have  more 
utility  than  the  remaining  two  OMs.  Its  rating  of  3?.  is  greater 
than  21+4  =  25;  hence,  the  utility  rating  for  moving-defense  does 
not  change.  The  final  action  is  to  rescale  the  final  set  of 
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revised  utility  ratings  to  obtain  utility  scores  (.50,  .28,  .18, 
.04).  A  similar  process  is  used  to  obtain  the  utility  scores  for 
Training  Management.  The  complete  set  of  results  from  substep  two 
for  the  example  CIEA  is  given  as  follows: 


Worth  Dimension: 

Initial 

Scaled 

Ratings 

1 

2 

3  4 

Utilit 
5  Scores 

Readiness  Evaluation 

1. 

Moving-assault 

43 

57 

.50 

2. 

Moving-defense 

32 

32 

.28 

3. 

Moving-FF 

21 

21 

.18 

4. 

Stationary-FF 

4 

4 

.04 

Training  Management 

1. 

Moving-FF 

54 

94 

94 

120 

.67 

2. 

Stationary-FF 

27 

27 

40 

40 

.22 

3. 

Zero  rifle 

14 

14 

14 

14 

.08 

4. 

Reduce  stoppage 

3 

3 

3 

3 

.02 

5. 

Load/unload 

3 

3 

3 

3 

.02 

Generate  Systems-Versus-Crlteria  Array 

Following  the  generation  of  importance  weights  for  WDs  and  utility 
scores  for  OMs  nested  within  WDs,  the  next  phase  in  MAUM-CIEA  involves 
generating  the  systems-versus-criteria  array.  In  CIEA,  the  systems-versus- 
criteria  array  is  a  matrix  that  presents  each  alternative  and  its  associated 
evaluation  criteria — Information  Utility,  Relative  Information  Utility, 
Information  Cost,  Relative  Information  Cost,  and  Relative  Information  Worth. 

Developing  the  systems-versus-criteria  array  is  a  five  step  proceuui.e, 
with  the  steps  listed  as  follows: 

1.  Obtain  System  Effectiveness  Ratings 

2.  Determine  Partial  Utilities  for  WDs 

3.  Aggregate  Partial  Utility  Scores 

4.  Estimate  Costs  of  Alternatives 

5.  Determine  Relative  Information  Worth 

Each  of  these  steps  is  discussed  in  the  following  paragraphs. 


2-22 


Obtain  Systems  Effectiveness  Ratings.  Within  the  context  of  CT.EA, 
system  effectiveness  is  defined  as  the  degree  to  which  an  alternative  pro¬ 
vides  timely,  quality  information  on  the  performances  relevant  to  the  DORAC 
evaluation.  Specifying  system  effectiveness  is  carried  out  in  three  sub¬ 
steps:  First,  information  quality  ratings  are  obtained  for  each  alternative 

on  each  OM.  Next,  each  DORAC  alternative  is  evaluated  with  respect  to  the 
utility  of  the  frequency  with  which  performance  data  are  provided.  Finally, 
the  quality  ratings  and  frequency  utility  scores  are  combined  to  produce 
a  single  measure  of  the  effectiveness  of  each  alternative  on  each  OM  nested 
within  each  WD.  The  next  series  of  paragraphs  describe  the  effectiveness 
rating  procedure  in  additional  detail. 

Information  quality  is  defined  as  the  extent  to  which  an  alternative 
provides  trustworthy  information  relevant  to  a  particular  OM.  Also  con¬ 
sidered  as  part  of  information  quality  is  the  amount  of  information  pro¬ 
vided;  that  is,  the  number  of  relevant  performance  conditions  that  are 
addressed  by  the  alternative.  In  CIEA,  information  quality  ratings  are 
obtained  using  the  procedure  outlined  as  follows: 

1.  Order  the  DORAC  alternatives  from  "best"  to  "worst" 
according  to  the  degree  to  which  each  alternative  is 
capable  of  providing  quality  information  relevant  to 
the  OM  under  consideration.  Factors  that  should  be 
considered  in  making  quality  judgments  include: 

a.  Amount  of  information.  The  number  of  relevant 
performance  conditions  that  are  addressed. 

b.  Dependability.  The  judged  trustworthiness  of  the 
data.  This  is  obtained  from  the  dependability 
ratings  made  previously  (e.g..  Figure  2-6). 

Ties  are  permitted.  If  one  or  more  of  the  alternatives 
are  judged  equivalent  in  terms  of  the  quality  of  the 
information  they  provide,  assign  them  the  same  rank. 

2.  Numerically  position  the  best  and  worst  alternatives  on  a 
O-to-100  scale.  Use  the  following  benchmark  ratings  as  a 
guide : 

0  -  The  alternative  provides  no  data  relevant  to  the 
OM  under  consideration. 

25  -  Marginal.  The  alternative  provides  partial  data 

on  the  OM  and  the  data  are  likely  to  be  undependable 
(e.g.,  the  recording/scoring  method  is  poor  re¬ 
sulting  in  low  validity  or  low  reliability). 
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50  -  Adequate.  The  alternative  provides  the  required 

data  but  some  dependability  problems  are  apparent. 

For  example,  the  most  appropriate  recording/scoring 
method  is  not  used  or  the  data  are  likely  to  have 
only  moderate  reliability. 

75  -  Good.  The  alternative  provides  required  data  in 
an  acceptable  manner.  Recording  methods  are 
acceptable;  reliability  is  likely  to  be  reasonably 
high. 

100  -  Excellent.  The  alternative  is  the  best  possible , 
given  the  current  technical  state  of  the  art. 

Recording  methods  are  automated  and  precise; 
reliability  is  likely  to  be  very  high. 

Position  the  remaining  alternatives  between  the  best  and  worst 
cases  on  the  O-to-100  scale.  Again,  refer  to  the  benchmark 
rating  points  presented  in  (2)  as  a  guide. 


Continuing  with  the  example  CIEA,  quality  ratings  for  the  three  hypo- 
th-  ticul  DORAC  alternatives  are  presented  in  Table  2-1. 

Table  2-1.  Quality  Ratings  for  Hypothetical 
DORAC  Alternatives 


Operational 

Measure 


Readiness  Evaluation 
Moving-Assault 
Mov :  -Defense 

Mj  .  iiig-FF 
Stationary-FF 
Training  Management 
Mov5n  ' 

Stat  _.jnary — FF 
Zero  riile 
Stoppage 
Load/Unload 


Alternative 


B 

X 

Y 

0 

0 

25 

0 

0 

50 

0 

0 

50 

50 

50 

75 

0 

25 

25 

25 

25 


0 

50 

50 


50 

40 

50 

25 

25 


25 

25 


After  information  quality  ratings  have  been  assigned  for  each  alterna 
tive  on  each  OM,  the  next  substep  in  obtaining  effectiveness  ratings  is  to 
determine  the  utility  of  the  evaluation  frequency  associated  with  each 
alternative.  Frequency  utility  ratings  are  obtained  by  applying  the 
following  sequence  of  actions: 

1.  Consider  the  frequency  of  the  information  provided  by 
each  alternative  (e.g.,  quarterly,  twice  a  year,  yearly, 
etc.).  Now,  specifically  considering  the  highest  and 
lowest  frequencies,  rate  the  usefulness  of  receiving 
DORAC  generated  information  with  the  frequencies  indicated. 
Use  a  O-to-100  scale  in  assigning  these  ratings. 

2.  Next,  consider  the  remaining  intermediate  frequencies. 
Position  the  remaining  frequencies  between  the  extreme 
values  (i.e.,  ratings  for  the  highest  and  lowest  fre¬ 
quencies)  on  the  O-to-100  scale. 

3.  Connect  the  scaled  points  with  a  line.  If  the  utility 
rating  for  the  lowest  frequency  is  not  zero,  connect  the 
point  associated  with  the  lowest  frequency  with  the  zero 
point  on  the  frequency-utility  axes.  The  result  is  the 
evaluation  frequency  utility  curve  for  the  alternatives 
under  consideration. 


As  an  example  of  the  frequency  utility  rating  process,  consider 
the  evaluation  frequencies  associated  with  the  DORAC  alternatives 
in  the  example  CIEA.  The  frequencies  for  alternatives  B,  X,  and  Y 
are  1,  2,  and  A  times  per  year,  respectively.  One  (the  lowest 
value)  is  assigned  a  utility  rating  of  50;  four  (the  highest  value) 
is  assigned  a  score  of  75.  The  only  intermediate  value,  two,  is 
assigned  a  rating  of  70.  The  resulting  utility  curve  for  evaluation 
frequency  is  shown  in  Figure  2-8. 


The  final  substep  in  the  process  of  obtaining  effectiveness  ratings 
for  alternatives  is  to  combine  the  quality  and  frequency  ratings  into  a 
single  measure  of  system  effectiveness.  Quality  and  frequency  are  combined 
using  the  formula; 


where  e..,  represents  the  effectiveness  of  the  i  DORAC 
^  alternative  on  the  k  ^  OM  nested  within  the 


Q.  is  the  quality  of  data  provided  by  the  i*”  ' 

alternative  on  the.  OM  nested  within  the  j 

and  F.  is  the  frequency  utility  rating  associated  with 
the  i^  DORAC  alternative. 


The  result  of  the  effectiveness  evaluation  procedure  is  a  systems 
effectiveness  matrix  for  each  WD.  Figure  2-9  presents  an  example  of  a 
hypothetical  systems  effectiveness  matrix  for  j •  The  cell  entries, 
represent  the  judged  effectiveness  scores  for  DORAC  alternatives  on  the 
j  I"®.  If  desired,  effectiveness  scores  for  all  alternatives  can  be 
combined  across  WDs  to  produce  a  systems  effectiveness  array.  For  exampl 
Table  2-2  presents  the  systems  effectiveness  array  for  the  example  CIEA. 


Table  2-2.  Systems-Ef fectiveness  Array  for 
Hypothetical  DORAC  Alternatives 


Operational 

Measure 


Readiness  Evaluation 

Moving-Assault 

Moving-Defense 

Moving-FF 

Stationary-FF 

Training  Management 

Moving-FF 

Stationary-FF 

Zero 

Stoppage 

Load/Unload 


Alternative 


B 

X 

Y 

0 

0 

18.75 

0 

0 

37.50 

0 

0 

37.50 

25.00 

35.00 

56.25 

0 

0 

37.50 

12.50 

35.00 

30.00 

12.50 

35.00 

37.50 

12.50 

17.50 

18.75 

12.50 

17.50 

18 . 75 

Figure  2-9.  Hypothetical  Systems  Effectiveness  Matrix 


Determine  Partial  Utilities  for  WD:^.  Following  the  generation  of 
the  systems  effectiveness  array,  the  next  step  in  the  analysis  is  tc 
produce  partial  utility  scores  for  each  alternative  on  each  WD.  The  gen¬ 
eral  form  of  the  equation  for  aggregating  effectiveness  ratings  across 
OMs  is  given  in  (2-3): 


f[u(Ky)]  -  E 


(2-3) 


th 


where  f[*]  is  the  partial  utility  score  of  the  i 
alternative  on  the  j  WD, 

is.  the  utility  score  of  the  OM  nested  in  the 

Jk  „D, 

and  is  the  ef  fecclvenesSj_rating  of  the  i*" '  DORAC 

alternative  on  the  k  *  OM  nested  within 
the  WD. 


Again  continuing  the  CIEA  example,  (2-3)  is  used  to  combine  the 
utility  scores  for  OMs  into  partial  utility  scores  for  WDs.  For 
example,  the  partial  utility  score  of  alternative  B  (baseline)  for 
Readiness  Evaluation  would  be  derived  as: 

u  =  (.50)(0)  +  (.28)(0) 

+  (.18)(0)  +  (.04)(25)  «  1.0 

(2-4) 

The  complete  set  of  partial  utility  scores  for  the  hypothetical 
DORAC  alternatives  is  presented  in  the  table  below. 

Alternative 

Worth  Dimension 

B  X  Y 

Readiness  Evaluation 

1.0  1.4  28.9 

Training  Management 

4.3  11.2  35.5 

Aggregate  Partial  Utility  Scores.  The  next  step  in  generating  the 
systems-versus-criteria  array  is  to  aggregate  partial  information  utility 
scores  across  WDs  to  produce  a  global  information  utility  score  for  each 
alternative.  In  accord  with  the  discussion  presented  in  Section  1,  an 
additive  utility  aggregation  model  is  recommended.  An  equation  for  pro¬ 
ducing  global  information  utility  scores  from  partial  information  utilities 
is  presented  as  follows: 
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(2-5) 


lU 


i 


I  W^f[u(x^j)], 


where 


and 


ID,  represents  the  global,  information  utility  score 
for  the  i  '  DORA.C  alternative, 


”3 

£[•] 


th 

is  the  importance  weiglit  of  the  j 

is  the  partial  Information  utility 
i  alternative  on  the  j  '  WD. 


WD, 

score  of 


the 


Following  (2-5),  the  Information  utility  scores  for  the  three 
hypothetical  DOIcAC  alternatives  are  obtained  as  follows: 

Alternative 

lUi 

B 

1.66 

X 

3,36 

Y 

30.22 

As  an  example  of 
line  alternative 

the  computation 
(B)  is  obtained 

of  III,  the  score  for  the 
as  follows: 

base- 

(.80)(1.0)  +  (.20)(A.3)  =  1.66 

(2-6) 

Estimate  Costs  of  Alternatives.  The  next-to-the-last  step  in  de¬ 
fining  the  systems-versus-criteria  array  involves  estimating  the  cost  of 
each  DORAC  alternative.  Although  costing  is  formally  introduced  at 
this  point  in  the  discussion,  the  cost  analysis  actually  may  be  initiated 
any  time  after  the  alternatives  have  been  specified  (i.e.,  following 
phase  A,  Develop  Alternative  System  Concepts).  The  cost  analysis  should, 
in  fact,  be  initiated  as  early  as  possible  since  this  aspect  of  CIEA  will 
usually  prove  to  be  the  most  time-consuming  component. 

The  objective  of  the  cost  analysis  substep  is  to  produce  a  cost  esti 
mate  for  each  alternative,  denoted  C..  To  assist  in  the  development  of 
cost  estimates  for  alternatives.  Appendix  A  presents  a  structured  DORAC 
Costing  Guideline.  The  guideline  leads  an  analyst  through  the  steps  of  a 
DORAC  cost  analysis  beginning  with  a  determination  of  the  anticipated 
facility  load  and  ending  with  a  total  estimated  cost  for  each  alternative 
It  should  be  noted  that  the  cost  estimates  provided  by  the  costing  guide¬ 
line  consider  only  those  design,  development  (e.g.,  testing  to  validate 
measures  and  establish  standards),  and  administration  (e.g.,  testing, 
data  processing)  costs  wiiich  would  be  incurred  over  and  above  those  asso¬ 
ciated  with  design,  development,  and  use  of  the  devices  for  training. 
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Determine  Relative  Information  Worth.  Within  the  standard  context 
of  cost-effectiveness  analysis  (e,g.,  Barish  &  Kaplan,  1978;  TRADOC 
Pamphlet  11-8),  Relative  Information  Worth  (RIW)  is  obtained  by  determining 
Relative  Information  Utility  (RIU)  and  Relative  Information  Cost  (RIC)  for 
each  alternative  and  then  combining  them.  RIU  for  alternative  ^  (RIU.) 
is  determined  by  dividing  the  information  utility  measure  for  alternative 
^  (lU.)  by  that  of  another,  usually  the  baseline  alternative  (i.e.,  the 
minimal  or  least  costly  DORAC  alternative): 

RIU.  =  lU./m.  (2-7) 

1  lb 

Relative  Information  Cost  for  alternative  (RIC.)  is  determined  in  a 
similar  fashion: 


RIC.  =  C./C.,  (2-8) 

lib’ 

tb 

where  C^  is  the  estimated  cost  of  the  i  alternative  and  C^  is  the  esti¬ 
mated  cost  of  the  baseline  alternative. 

RIU  and  RIC  are  useful  in  themselves,  but  have  limited  value  in  the 
context  of  an  analysis  similar  to  CIEA  where  the  objective  is  to  identify 
the  most  cost-effective  alternative.  In  order  to  determine  a  preferred 
capability,  RIU  and  RIC  are  integrated  into  a  measure  of  RIW  for  each 
alternative.  In  CIEA,  the  RIW  of  the  i  alternative  (RIW.)  is  defined 
as  follows: 

RIW.  =  RIU. /RIC.  =  ^^i^^^b  .  (2-9) 

An  RIW  score  greater  than  one  indicates  that  alternative  is  preferred 
to  the  baseline  system.  In  effect,  what  is  done  is  to  normalize  system 
cost  and  information  utility  relationships  with  the  baseline  alternative 
assigned  a  unit  value. 


To  Illustrate  the  development  of  RIW,  again  consider  the 
three  hypothetical  DORAC  alternatives.  Assume  that  cost 
estimates  for  the  three  alternatives  have  been  determined  as 
follows : 


Alternative 


C 

Estimated  Cost —  i 


B 

X 

Y 


$  175,000 
525,000 
945,000 
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Applying  (2-7)  and  (2-8),  the  following  RIC  and  RIU  scores  are 
obtained : 


Alternative 

B 

X 

Y 


RIU 

1.00 

2.10 

18.20 


Finally,  using  (2-9),  RIW  for  each  of  the  hypothetical  DORAC 
alternatives  is  determined: 

Alternative  RIW 

B  1.00 

X  0.70 

Y  3.37 

Based  on  RIW  alone,  the  results  indicate  that  alternative  Y 
is  the  most  cost  and  information  effective  capability. 


As  a  final  step  in  phase  6,  all  of  the  system  evaluation  criteria 
can  be  assembled  into  a  formal  systems-versus-criteria  array.  In  this 
array,  the  rows  represent  DORAC  alternatives  and  the  columns  represent 
the  various  system  evaluation  criteria  (e-o-.,  lU,  RIU,  IC ,  RIC,  and  RIW) 
The  entries  in  the  matrix  are  the  appropriate  evaluation  criteria  classi 
fied  by  alternative.  As  an  example,  the  following  table  is  the.  formal 
systems-versus-criteria  array  for  the  set  of  hypothetical  DORAC  alterna¬ 
tives. 


Evaluation  Criterion 

Alternative 

lU 

RIU 

IC 

RIC 

RIW 

B 

1.66 

1.00 

175,000 

1.00 

1.00 

X 

3.36 

2.10 

525,000 

3.00 

0.70 

Y 

30.22 

18.20 

945,000 

5.00 

3.37 

Perform  Sensitivity  Analyses 


It  is  likely  that  many  of  the  values  used  in  ClEA  to  specify  lU  or 
system  cost  will  be  based  upon  assumption,  expert  opinion,  or  incomplete 
data  and  thus  be  of  unknown  validity.  Sensitivity  analysis  refers  to  an 
investigation  of  the  effects  on  system  evaluation  criteria  of  estimated 
parameters  taking  on  values  different  from  those  used  in  the  analysis 
(Shannon,  1975).  Such  analyses  usually  consist  of  systematically  varying 
the  values  of  selected  decision  variables  over  a  range  of  interest  and 
observing  the  effects  of  these  changes  on  system  evaluation  criteria. 

Sensitivity  analysis  can  indicate  the  robustness  of  the  results  of 
a  CIEA.  It  is  desirable  to  determine  how  far  off  certain  parameter  esti¬ 
mates  can  be  without  changing  the  basic  conclusions  of  the  analysis.  If 
the  results  are  insensitive  to  a  fairly  wide  range  of  changes  in  selected 
parameters,  then  excessii'e  concern  need  not  be  given  to  the  accuracy  of 
these  parameters.  On  the  other  hand,  if  the  results  prove  to  be  highly 
dependent  on  the  values,  of  certain  parameters,  it  may  be  prudent  to  ex¬ 
pend  additional  resources  and  obtain  more  precise  estimates  for  the 
parameters  in  question. 

In  CIEA,  one  candidate  for  sensitivity  investigation  is  the  effect 
of  estimated  system  cost  on  the  selection  of  a  preferred  alternative. 

The  first  step  in  the  conduct  of  a  cost  sensitivity  analysis  is  to  estab¬ 
lish  the  cost  range  over  which  the  top-rated  alternative  is  preferred; 
that  is;  to  determine  the  cost  estimate  that  would  make  the  top-rated  al¬ 
ternative  no  longer  preferred.  Then,  determine  a  pessimistic  (i.e.,  high¬ 
est)  cost  estimate  for  the  preferred  alternative.  The  pessimistic  cost 
estimate  can  be  obtained  by  reverting  to  the  cost  determination  portion 
of  the  CIEA  and  using  pessimistic  instead  of  expected  values  in  the  cost 
analysis.  Next,  compare  the  pessimistic  cost  estimate  with  the  upper 
bound  of  the  cost  range  where  the  top-rated  alternative  is  preferred. 

If  the  pessimistic  cost  estimate  is  below  the  upper  cost  bound,  the  top- 
rated  alternative  is  still  preferred.  However,  if  the  pessimistic  cost 
estimate  is  above  the  upper  cost  bound,  then  the  highest  rated  DORAC 
alternatives  should  be  examined  in  greater  detail  before  selecting  a 
preferred  capability. 


To  illustrate  the  cost  sensitivity  analysis  procedure  outlined 
above,  consider  the  top-iated  hypothetical  DORAC  al*-ernative .  Al¬ 
ternative  Y  has  an  RIU  score  of  18.20,  an  RIC  score  5.40,  and  a 
resulting  RIW  score  of  3.37.  The  second  rated  alternative, 
baseline  (B) ,  has  RIW,  RIU,  and  RIC  scores  of  1.00.  Accordingly, 
the  RIC  score  for  alternative  Y  would  have  to  exceed  18.20  for 
Y's  RIW  score  to  be  less  than  that  of  alternative  B.  In  other 
words,  alternative  Y's  estimated  cost  would  have  to  exceed 
$175,000  X  18.20  =  $3,185,000  before  the  results  of  the  analysis 
should  be  reconsidered. 
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Situations  may  also  arise  in  which  importance  ratings  for  WDs,  utility 
scores  for  OMs,  or  effectiveness  ratings  for  DORAC  alternatives  are  at 
issue;  that  is,  varied  opinions  concerning  the  values  of  certain  of  these 
parameters  are  apparent.  An  approach  to  resolving  these  differences  is  to 
conduct  a  sensitivity  investigation  generally  knowii  as  parametric  analysis. 
In  a  parametric  analysis,  the  values  of  the  parameters  in  question  are 
systematically  varied  over  a  range  of  interest  (i.e.,  the  range  dictated 
by  different  value  structures).  The  effects  of  these  variations  on  system 
sei.ection  criteria  are  observed.  If  the  parametric  analysis  indicates 
that  different  parameter  values  (worth  values)  reflecting  different  points 
of  view  result  in  changes  in  system  rankings,  then  a  decision  must  be  made 
concerning  which  value  structure  is  to  dominate. 


Select  Preferred  Alternative 


The  last  phase  in  CIEA  is  to  select  a  preferred  DORAC  alternative 
from  among  those  under  consideration.  In  many  situations,  the  decision 
rule  is  simple:  maximize  RIW,  the  index  of  Relative  Information  Worth. 
This  choice  should  be  made  after  reviewing  the  results  of  appropriate 
sensitivity  analyses. 

If  the  RIW  scores  for  a  number  of  alternatives  are  virtually  identi¬ 
cal,  the  appronriate  course  of  action  may  be  to  conclude  that  there  is  no 
difference  among  the  top-rated  alternatives.  In  such  situations,  the  se¬ 
lection  of  a  preferred  DORAC  alternative  may  have  to  be  made  on  the  basis 
of  additional  analyses  or  on  the  basis  of  considered  military  judgment. 

As  a  final  note,  a  special  selection  case  arises  when  one  of  the 
evaluation  criteria  is  subject  to  a  constraint  (e.g.,  a  budget  constraint 
exists  or  the  lU  score  for  an  alternative  is  judged  unacceptable).  The 
decision  rule  to  maximize  RIW  is  easily  adapted  to  situations  involving 
maximum  or  minimum  acceptable  values  on  a  given  criterion.  When  such 
situations  arise,  simply  eliminate  alternatives  that  violate  the  con¬ 
straint,  regardless  of  their  overall  RIW  scores. 


SECTION  3 


CIEA  DEVELOPMENTAL  ISSUES 
Recap 


The  previous  sections  have  presented  an  overview  of  the  problem 
situation  addressed  by  CIEA;  i.e.,  determining  the  point  at  which  the 
cost  of  obtaining  the  readiness  information  provided  by  a  DORAC  is  off¬ 
set  by  the  information's  value  to  recipients.  An  approach  to  assessing 
information  worth  based  on  MAUM  was  also  presented  and  illustrated  with 
a  hypothetical  example.  The  MAUM  approach  to  information  worth  assess¬ 
ment  is  illustrated  in  greater  detail  in  the  report  Training  Device 
Operational  Readiness  Assessment  Capability  (DORAC) ;  Feasibility  and 
Utility  (Hawley  &  Dawdy,  1980).  The  companion  report  presents  the  re¬ 
sults  of  an  actual  CIEA  performed  on  a  set  of  training  devices  for  the 
M-16A1  rifle. 


As  noted  in  Section  1,  lU,  the  output  of  the  MAUM-based  evaluation 
procedure,  is  used  as  a  proxy  measure  for  strict  IW.  The  rationale  for 
this  approach  is  based  upon  the  difficulty  of  assessing  IW  using  other 
potential  methods,  such  as  decision-maker  action  substitution.  Using  lU 
as  a  proxy  measure  for  IW  is  based  on  the  assumption  that  IW  is  an  in¬ 
creasing  monotonic  function  of  lU;  that  is, 

IW  =  <I>(IU),  (3-1) 


such  that  IW^  _<  IW.  whenever  lU.  <_  lU..  In  other  words,  as  lU  increases, 
IW  also  increases, ^or  at  least  Soes  not  decrease.  Figure  3-1  graphically 
illustrates  one  monotonic  example  of  the  assumed  monotonic  relationship 
expressed  in  (3-1) . 


Information 

Worth 


Figure  3-1.  Assumed  Monotonic  Relationship  Between 

Information  Worth  and  Information  Utility 


An  obvious  issue  to  be  resolved  before  a  widespread  acceptance  of 
the  results  of  MAUM-CIEA  is  the  vcuLixLity  of  the  assumption  expressed  in 
(3-1)  blishing  the  validity  of  using  lU  as  a  proxy  measure  for  IW 

is  b£^^  \  _ly  a  two-step  procedure.  The  steps  involved  in  validating  the 
assumpL ion  are  listed  as  follows: 

1.  Establishing  that  the  MAUM-based  lU  assessment  procedure 
is  a  well-behaved  (i.e. ,  reliable)  process.  That  is, 
determining  whether  or  not  MAUM-derived  measures  of  lU 
are  consistent,  equal-interval  estimates  of  IW. 

2.  Establishing  that  IW  is  a  monotonically  increasing 
function  of  lU,  as  assessed  using  the  MAUM-based  assess¬ 
ment  procedure.  This,  essentially,  involves  establishing 
the  predictive  validity  of  the  MAUM-CIEA  methodology. 

Each  of  the  above  issues,  plus  related  methodological  Issues,  is  addressed 
in  more  detail  in  the  following  subsection. 


Methodological  Issues 

The  objective  of  the  lU  assessment  portion  of  the  CIEA  methodology 
is  to  provide  a  measure  of  the  "value  for  decision-making"  of  a  given 
amount  of  information  obtainable  from  a  DORAC.  Stevens  (1951)  defines 
measurement  as  the  process  of  assigning  numbers  to  objects  or  events 
according  to  rules.  The  MAUM-CIEA  methodology  outlined  in  Section  2  de¬ 
fines  one  set  of  rules,  or  one  "yardstick",  for  measuring  IW.  However, 
the  usability  of  the  MAUM-based  yardstick  has  not  been  established.  To 
date,  little  Information  regarding  the  actual  properties  of  MAUM-based 
lU  scores  has  been  produced.  To  be  appropriate  for  use  in  CIEA,  it  is 
necessary  to  demonstrate  that  the  MAUM  procedure  is  broadly  generalizable 
and  that  resulting  lU  scores  are: 

1.  Reliable, 

2.  Properly  scaled  (i.e.,  at  least  equal-interval),  and 

3.  Predict ively  valid  indices  of  strict  IW. 

A  primary  requirement  for  any  CIEA  IW  evaluation  procedure  is  that 
it  be  broadly  generalizable.  Within  the  context  of  DORAC  evaluation, 
generalizabllity  refers  to  the  methodology  being  usable  with  training  de¬ 
vices  of  varying  complexity  (i.e.,  ranging  from  a  few  performances/condi¬ 
tions  with  a  single  measurement  capability  to  a  large  number  of  performances/ 
conditions  with  a  sophisticated  measurement  capability)  at  various  stages  in 
their  developmental  cycle  (e.g.,  conceptual,  bread-board,  fielded).  The 
MAUM-CIEA  methodology  outlined  in  Section  2  has  been  demonstrated  using 
a  l^oJidzd  training  device  system  having  only  low  to  moderate  compexity. 


as  Indexed  by  the  number  of  relevant  performance  objectives  and  the  so¬ 
phistication  of  the  associated  measurement  capability  (see  Hawley  &  Dawdy, 
1980).  Hence,  the  first  methodological  issue  to  be  addressed  concerns 
establishing  that  the  MAUM-CIEA  methodology  is  generalizable  to  a  range 
of  training  devices.  This  issue  can  be  resolved  only  through  an  applica¬ 
tion  of  the  methodology  to  a  series  of  training  devices  spanning  the  De¬ 
vice  Complexity  -  Developmental  Stage  axes.  Through  a  se  'rs  of  such 
applications,  implementation  and  procedural  problems  can  be  identified 
and  rectified.  The  resulting  evaluation  data  can  also  be  used  in  addi¬ 
tional  aspects  of  the  validation  process,  such  as  investigating  reliability, 
scaling  properties,  and  predictive  validity. 

Once  it  has  been  established  that  the  MAUM-CIEA  procedure  is  opera¬ 
tionally  general!-  jle,  a  second  methodological  concern  is  the  reliability 
of  the  process.  In  general,  reliability  refers  to  the  aonsistenai)  from 
one  set  of  measurements  to  another  on  repetition  of  a  measurement  pro¬ 
cedure  (Stanley,  1971).  In  the  case  of  MAUM-CIEA,  reliability  denotes 
the  stability  or  reproducibility  of  lU  results  upon  repeated  application 
of  the  methodology  by  independent  sets  of  raters.  An  obviously  desirable 
state  of  affairs  is  that  CIEA  results  be  reasonably  independent  of  whom¬ 
ever  constitutes  the  decision-making  group,  given  that  equally  qualified 
representatives  of  the  same  stakeholders  provide  the  constituent  ratings. 

Proceeding  from  the  definition  of  reliability  presented  in  the  pre¬ 
vious  paragraph,  MAUM-CIEA  reli^ibility  will  likely  have  to  be  assessed 
in  a  manner  analogous  to  that  of  parallel  forms  reliability  in  psycho¬ 
logical  test  theory.  In  psychological  test  development,  parallel  forms 
reliability  is  established  by  first  developing  two  independent  testing 
procedures  (i.e.,  parallel  forms)  assumed  to  provide  the  same  true  score. 
Next,  each  form  of  the  test  is  administered  to  equivalent  groups  of 
testees.  The  correlation  of  results  obtained  using  the  two  testing  pro¬ 
cedures  provides  the  basis  for  computing  a  reliability  coefficient  (Lord 
&  Novick,  1968). 

In  the  case  of  MAUM-CIEA,  establishing  reliability  would  be  done  in 
a  conceptually  similar  fashion:  Independent  groups  of  decision-makers 
representing  the  same  stakeholders  would  complete  the  MAUM  procedure 
evaluating  the  same  set  of  DORAC  alternatives.  The  results  of  the  re¬ 
peated  applications  would  then  be  compared  at  a  variety  of  levels;  e.g., 
importance  weights  for  WDs,  partial  utility  scores  for  OMs,  effectiveness 
scores  for  alternatives,  and  aggregate  lU  scores.  The  degree  of  con¬ 
sistency  across  levels  would  provide  an  indication  of  the  reliability 
of  the  MAUM-CIEA  methodology.  Data  used  in  this  empirical  examination 
of  reliability  could  be  obtained  by  replicating  certain  of  the  evalua¬ 
tions  used  to  establish  generalizability .  For  example,  specified  analy¬ 
ses  could  be  conducted  on  a  single  set  of  alternatives  using  two  inde¬ 
pendent  sets  of  decision-makers. 
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Strictly  speaking,  the  reliability  assessment  procedure  outlined  in 
the  previous  paragraph  will  not  demonstrate  the  absolute  reliability  of 
the  MAUM  method,  but  Instead  only  the  results  of  its  application  in  a 
particular  situation  (e.g.,  for  a  training  device  of  a  given  complexity 
at  a  given  developmental  st;-,/-  a-.-d  for  a  given  group  of  decision-makers) 
.(Torgerson,  1958).  Hence,  it  would  be  desirable  to  demonstrate  experimen¬ 
tally  the  reliability  of  thr  M.VdM-CIEA  procedure  across  a  range  of  de¬ 
vices  to  which  the  meth':dology  might  be  applied.  This  could  be  accom¬ 
plished  through  a  replication  of  the  parallel  groups  evaluation  process 
across  a  range  of  training  devices.  In  any  event,  even  a  single  demon¬ 
stration  of  reliability  would  serve  to  enhance  user  confidence  in  the 
resulting  data. 

A  third  methodological  issue  relevant  to  the  acceptability  of  the 
MAUM-CIEA  IU  evaluation  procedure  concerns  the  scaling  properties  of  the 
resulting  data.  In  determining  utilities  using  any  of  the  variations  of 
the  Direct  scaling  methods,  it  is  assumed  that  decision-makers  are  capable 
of  rating  WDs,  OMs,  and  the  effectiveness  of  DORAC  alternatives  on  an 
equat-intevoal  subjective  scale.  If  this  assumption  is  correct,  then  the 
procedures  presented  in  Section  2  provide  scale  values  that  have  equal- 
interval  properties.  It  should  be  noted,  however,  that  the  scaling 
methods  themselves  provide  no  explicit  means  of  testing  this  assumption. 

The  assumption  that  decision-makers  are  capable  of  providing  equal- 
interval  scale  values  for  WDs,  OMs,  and  system  effectiveness  is  critical 
to  the  system  evaluation  procedures  used  in  MAUM-CIEA.  The  use  of  MAUM- 
derived  lU  scores  in  the  computation  of  RIU  and  RIW  is  based  upon  the 
assumption  that  the  level  of  measurement  for  lU  is  at  least  equal-interval 
(i.e.,  equal-interval  or  ratio).  The  effects  on  RIU  and  RIW  (and  thus 
the  eventual  selection  of  a  preferred  DORAC  alternative)  of  violations 
of  the  equal-interval  assumption  are  not  known.  A  reasonable  conclusion, 
however,  is  that  if  IU  is  at  most  ordinal,  then  RIU  and  RIW  are  at  most 
ordinal.  The  use  of  cost-effectiveness  ratios  (e.g.,  RIW)  is  based  on 
the  assumption  that  both  the  numerator  and  denominator  terms  are  at  least 
equal-interval.  Using  this  powerful  tool  for  integrating  system  cost  and 
effectiveness  measures  is  inappropriate  if  the  scaling  properties  of 
either  numerator  or  denominator  are  suspect.  In  fact,  the  undesirable 
effects  of  possible  violations  of  the  equal-interval  assumption  have  re¬ 
sulted  in  a  general  aversion  to  the  use  of  cost-effectiveness  ratios 
within  the  military  systems  analysis  community  (Paris,  1980). 

In  view  of  the  above  discussion,  the  validity  of  the  equal-interval 
assumption  should  be  examined  empirically.  As  in  the  case  of  general- 
izability  and  reliability,  testing  the  equal-interval  assumption  would 
require  a  repeated  application  of  the  MAUM-CIEA  procedure  in  the  same 
evaluation  situation.  Repeated  applications  are  necessary  because  data 
obtained  in  a  single  application  provide  no  basis  for  establishing 
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whether  or  not  decision-makers  are  judging  on  the  basis  of  an  equal- 
interval  scale  (Torgerson,  1958).  It  is  always  possible  to  compute  scale 
values  on  the  basis  of  an  equal-interval  assumption.  In  addition,  the 
consistency  of  judgments  across  groups  is  in  itself,  not  an  adequate 
criterion  by  which  to  evaluate  the  equal-interval  assumption.  Completely 
inconsistent  judgments  are  evidence  that  ratings  do  not  follow  an  equal- 
interval  scale.  Consistent  ratings,  however,  do  not  imply  that  decision¬ 
makers  are  judging  on  an  equal-interval  scale.  A  criterion  based  on 
consistency  alone  does  not  distinguish  between  equal-interval  judgments 
and  straight  ordinal  position  judgments. 

The  minimum  requirement  for  an  equal-interval  scale  is  that  the 
ratios  of  differences  in  scale  values  assigned  to  any  three  or  more  stim¬ 
uli  are  invariant  with  respect  to  the  values  of  the  remaining  stimuli  in 
the  set  (Torgerson,  1958).  This  can  be  experimentally  verified  by  plotting 
scale  values  obtained  from  one  evaluation  against  the  scale  values  from 
a  second  independent  replication  (easily  obtained  within  the  context  of 
replications  directed  at  establishing  generalizabllity  and  reliability). 

If  the  equal-interval  assumption  is  valid,  the  plot  will  be  linear,  within 
sampling  error.  Again,  as  in  the  case  of  reliability,  a  demonstration 
that  decision-makers  used  an  equal-interval  scale  in  one  situation  will 
not  necessarily  generalize  to  other  situations.  Repeated  demonstrations 
of  the  validity  of  the  equal-interval  assumption  would,  however,  build 
user  confidence  in  the  applicability  of  using  MAUM-derived  lU  scores, 
and  thus  RIW,  in  CIEA. 

A  fourth  methodological  issue,  somewhat  related  to  the  issue  of  the 
scaling  properties  of  lU  scores,  concerns  overall  methodological  complex¬ 
ity.  The  CIEA  methodology  presented  in  Section  2  employs  a  mixture  of  .de¬ 
composition  and  holistic  utility  evaluation  procedures  to  determine  lU. 
Decisions  concerning  the  use  of  decomposition  versus  holistic  judgments 
at  various  places  in  the  procedure  were  made  on  the  basis  of  previous 
MAUM  research  and  applications  and  on  the  basis  of  perceived  limits  on 
the  complexity  of  the  resulting  analytical  method.  Should  the  current 
version  of  the  MAUM-CIEA  methodology  not  provide  suitably  scaled  lU 
scores,  a  potential  means  of  raising  the.  level  of  measurement  for  lU 
is  to  examine  the  suitability  of  the  utility  evaluation  procedures  used 
in  the  analysis.  The  intent  of  this  examination  would  be  to  refine  the 
methodology  by  using  the  most  appropriate  evaluation  procedures;  that 
is,  by  using  decomposition  methods  where  they  are  most  appropriate  and 
holistic  methods  where  they  are  most  appropriate.  It  should  be  noted, 
however,  that  one  result  of  such  modifications  (e.g.,  replacing  a  holistic 
by  a  decomposition  procedure)  could  be  an  increased  level  of  complexity 
in  the  analysis  and,  thus,  an  elicitation  procedure  that  is  more  diffi¬ 
cult  for  decision-makers  to  employ. 
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The  MAUM-CIEA  methodology  outlined  in  Section  2  is  already  relatively 
complex  in  terms  of  process  branching,  the  number  of  judgments  required, 
and  the  information  load  placed  on  users.  As  with  most  MAUM  procedures, 
MAUM-CIEA  also  tends  to  be  time  consuming  in  its  application.  Increasing 
the  complexity  of  the  method  beyond  the  present  level  will  act  against 
the  widespread  applications  of  MAUM-CIEA  as  part  of  the  DORAC  evaluation 
process. 

One  way  to  alleviate  the  demands  placed  on  MAUM-CIEA  users  is  to 
computer-aid  the  elicitation  process.  Computer-aiding  the  elicitation 
process  would  involve  developing  an  Interactive  computer  capability  to 
guide  and  assist  decision-makers  in  applying  the  methodology.  Several 
studies  have  indicated  that  computer  aiding  MAUM-based  decision  processes 
can  result  in  time  savings  as  well  as  enhanced  decision  consistency 
(Freedy,  David,  Steeb,  Samet,  &  Gardiner,  1976;  Sicherman,  1975;  Ulvila, 
1975). 

A  fifth  methodological  issue  relevant  to  the  application  of  MAUM- 
CIEA  concerns  the  predictive  validity  of  lU  scores;  that  is,  establishing 
that  lU  can,  in  fact,  be  used  as  a  proxy  measure  for  strict  IW.  As  noted 
previously,  lU  is  used  as  a  proxy  measure  for  IW  under  the  assumption 
that  IW  is  an  increasing  monotonlc  function  of  lU.  Hence,  establishing 
that  lU  can  be  used  in  lieu  of  strict  IW  reduces  to  the  problem  of  vali¬ 
dating  the  assumption  of  monotonicity. 

Given  the  current  status  of  CIEA  methodological  development,  the 
assumption  of  monotonicity  will  likely  have  to  be  tested  within  a  con¬ 
vergent  validation  framework  (Campbell  &  Fiske,  1959).  In  the  present 
situation,  convergent  validation  would  involve  determining  the  extent 
to  which  lU  (obtained  using  MAUM)  is  related  to  other  independently  de¬ 
rived  measures  of  IW.  Hence,  the  first  requirement  in  validating  the 
use  of  lU  as  a  proxy  measure  for  strict  IW  involves  developing  one  or 
more  independent  methods  for  assessing  IW. 

At  the  present  time,  IW  is  rather  loosely  defined  as  "value  for 
decislon-rmaking" .  Before  developing  alternative  methods  for  assessing 
IW,  it  will  first  be  necessary  to  define  IW  in  exact  terms.  Carnap 
(1950)  has  referred  to  this  process  as  explication ^  or  operational 
definition.  Along  with  the  operational  definition,  rules  for  the  meas¬ 
urement  of  the  construct  are  also  specified.  Then,  given  an  operational 
definition  for  IW  and  rules  for  its  measurement,  the  next  step  in  vali¬ 
dating  MAUM-CIEA  is  to  assess  IW  using  one  or  more  independent  procedures. 
Agreement  between  lU,  as  obtained  using  MAUM-CIEA,  and  the  other  inde¬ 
pendent  measures  of  IW  would  lend  support  to  the  use  of  lU  as  a  proxy 
measure  for  IW. 

One  approach  that  may  hold  promise  for  developing  an  alternative 
measure  of  IW  involves  the  use  of  combat  simulation  models  (CSMs)  (e.g.. 
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ASARS,  COTEAM,  CARMONETTE,  etc.).  Conceptually  speaking,  CSMs  have  the 
potential  for  determining  the  nature  of  the  relationship  between  certain 
aspects  of  individual  or  collective  performance  (e.g.,  M-16  A1  accuracy)  and 
overall  individual  or  unit  combat  effectiveness.  That  is,  CSMs  may  have 
potential  for  specifying  (at  least  partly)  the  nature  of  the  production 
function  expressed  as  (3-2) : 

z  =  ^2’  ^^3’  ’^n^*  (3-2) 

where  £  represents  individual  or  unit  combat  effectiveness, 
the  represent  the  level  of  performance  on  component  i^, 
and  f(‘)  is  a  function  relating  performance  levels  to  £. 


Should  it  prove  possible  to  form  (3-2),  it  then  reasonably  follows 
that  IW  could  be  defined  as  a  function  of  the  parameters  relating  per¬ 
formance  levels  to  £.  For  example,  if  changes  in  M-16  A1  accuracy,  do  not 
affect  £,  then  Information  concerning  M-16  A1  accuracy  has  little  real  value 
whereas,  in  the  same  situation,  if  fire  distribution  patterns  are  found 
to  impact  significantly  on  £,  then  information  concerning  fire  distribu¬ 
tion  patterns  h?s  high  value.  Mathematically,  the  relationship  between 
the  parameters  relating  the  x^  to  £  and  IW  can  be  expressed  as  follows; 

TW  =  g(o^,  oi2>  £3> 

where  IW  represents  Information  worth, 

the  a.  represent  the  parameters  in  f(*)  relating  the 
x^  to  £, 

and  g(*)  is  a  function  relating  the  to  IW. 


Short  of  assessing  results  of  actual  (or  near  actual)  combat  situa¬ 
tions,  attempting  to  evaluate  before  and  after  decision  scenarios  (see 
the  discussion  presented  in  Section  1),  or  arbitrarily  defining  lU  to 
be  an  acceptable  measure  of  strict  IW,  the  CSM  approach  to  developing 
an  alternate  measure  of  IW  is  the  only  viable  means  for  establishing 
the  predictive  validity  of  MAUM-CIEA.  Relying  on  user  opinions  regarding 
the  reasonableness  (i.e.,  face  validity)  of  MAUM-CIEA  results  is  not 
sufficient.  Face  validity  can  be  unrelated  to  predictive  or  criterion- 
referenced  validity  (Lord  &  Novick,  1968).  The  MAUM-CIEA  methodology 
provides  a  means  for  assessing  the  perceptual  set  of  a  group  of  decision¬ 
makers.  If  the  perceptual  set  is  in  error,  then  all  an  additional 
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round  of  subjective  evaluations  will  accomplish  is  to  validate  existing 
institutional  prejudices.  As  with  other  evaluative  methodologies,  face 
validity  is  important  to  the  acceptance  of  results.  In  the  final  analysis, 
however,  the  results  of  MAUM-CIEA  should  be  exposed  at  least  once  to  the 
acid  test  of  objective  reality,  even  if  that  reality  is  determined  in  a 
somewhat  contrived  or  limited  situation. 

The  previous  points  should  not  be  taken  to  imply  that  CSM-CIEA  (or  a 
variant)  is  always  to  be  preferred  to  MAUM-CIEA.  Given  that  IW  is  demon¬ 
strated  to  be  monotonically  related  to  lU,  MAUM-CIEA  will  be  preferred  in 
most  situations.  For  example,  appropriate  CSMs  may  not  exist  or  the  cost 
of  CSM-CIEA  might  preclude  its  practical  application.  It  should  be  empha¬ 
sized  that  the  application  of  CSM-CIEA  is  likely  to  be  quite  expensive 
relative  to  an  application  of  MAUM-CIEA.  The  point  remains,  however,  that 
the  predictive  validity  of  MAUM-CIEA  is  a  critical  issue  and  should  be 
empirically  evaluated,  to  the  extent  deemed  practical  and  sufficient. 

An  ancillary  result  of  the  successful  development  of  a  CSM-CIEA 
methodology  would  be  the  potential  for  specifying  more  Vei'idioat  per¬ 
formance  standards  for  the  performances  relevant  to  the  DORAC  under  study. 
Veridical  performance  standards  are  defined  as  those  that  are  based  upon 
the  true  contribution  of  individual  performances  to  combat  effectiveness. 

If  the  function  expressed  as  (3-2)  can  be  specified,  then  a  potential 
exists  for  establishing  realistic  standards  for  the  performances  Involved 
in  the  x^.  As  in  personnel  selection,  the  first  step  in  such  a  develop¬ 
mental  process  would  be  to  specify  a  minimally  acceptable  level  for  £. 

Then,  the  minimum  levels  for  various  x,  required  to  produce  the  minimum 
level  of  £  could  directly  be  determined.  Interactions,  or  trade  offs, 
between  levels  of  x^  and  £  could  also  be  studied.  The  resulting  data 
could  prove  useful  in  the  conduct  of  CTEA  and  other  training  development 
or  evaluation  activities. 

In  closing,  this  report  has  presented  a  methodology,  MAUM-CIEA,  for 
establishing  the  cost-effectiveness  of  Information  provided  by  potential 
DORACs.  As  noted  earlier  in  this  section  of  the  report,  several  method¬ 
ological  Issues  relevant  to  MAUM-CIEA  remain  points  of  inquiry.  This 
situation  is,  however,  not  unusual  in  the  case  of  newlv  developed  analyt¬ 
ical  procedures,  even  those  that  are  integrated  from  proven  components. 

The  Issues  noted  in  this  section  should  be  explored.  Exploring  them  will 
take  time,  but  in  the  Interim  MAUM-CIEA  need  not  remain  on  the  shelf.  If 
MAUM-CIEA  results  are  viewed  cautiously  until  more  data  regarding  the 
me  hodological  issues  are  avaulable,  the  methodology  can  (and  should) 
be  used  in  the  evaluation  of  potential  DORACs  or  in  other  potentially- 
relevant  aspects  of  the  training  device  development  process.  Data  ob¬ 
tained  from  such  Initial  applications  of  MAUM-CIEA  could  prove  highly 
useful  in  streamlining  the  procedure  and  in  making  the  method  more  re¬ 
sponsive  to  actual  user  needs. 
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APPENDIX  A 


DORAC  COSTING  GUIDELINE 
Introduction 


The  objective  of  the  following  costing  guideline  is  to  lead  an 
analyst  through  a  series  of  steps  that  will  lead  to  an  estimate  of  the 
cost  of  developing,  acquiring,  and  operating  a  DORAC.  Factors  that  are 
considered  in  estimating  the  total  cost  of  the  capability  include  the 
following : 

1.  Facilities — Physical  assets  (buildings,  land,  etc.) 
required  for  the  implementation  and  support  of  a 
DORAC  system. 

2.  Equipment — Hardware  directly  associated  with  evaluation 
positions  and  their  support. 

3.  Materials — Software  (user  materials,  films,  texts, 
computer  programs,  etc.)  necessary  for  the  conduct 
and  support  of  DORAC  evaluations. 

4.  Personnel — Evaluator  and  evaluatee  costs  incurred  as  a 
direct  result  of  DORAC  evaluation  activities  (e.g., 
salaries,  travel,  per  diem,  expendable  supplies,  etc.). 

The  first  step  in  costing  a  DORAC  is  to  specify  the  exact  nature  of 
each  alternative;  that  is,  to  list  the  devices  required,  the  planned 
number  of  evaluations  per  year,  and  support  requirements  in  terms  of 
operators,  evaluators,  and  maintenance  personnel;  and  so  forth.  Then, 
for  each  training  device  comprising  part  of  an  alternative  and  for  the 
baseline  alternative,  the  following  series  of  steps  is  completed: 

Step  1.  Compute  Anticipated  Facility  Load  (AFL) 

a.  Specify  Annual  Evaluation  Frequency  (AEF) ,  the  number  of  times 
each  evaluatee  must  be  evaluated  annually  using  the  device. 

b.  For  the  device,  specify  the  Evaluation  Cycle  Size  (ECS),  the 
number  of  evaluation  positions  that  can  be  used  simultaneously. 

c.  Identify  Organization  Evaluation  Unit  Size  (OEUS) ,  the 
evaluatee  group  size  (e.g.,  individual,  squad,  platoon, 
company,  etc.)  organic  to  the  base  organization  (e.g., 
battalion,  brigade,  division,  post,  etc.)  that  most  closely 
matches,  without  exceeding,  the  ECS. 


d.  Specify  the  number  of  Evaluatees  per  Organization  (E/O),  the 
total  number  of  personnel  within  the  base  organization  that 
the  evaluation  is  applicable  to. 


e.  Compute  Organizational  Evaluations  (OE) ,  the  number  of 

evaluation  cycles  required  to  process  one  base  organization: 


OE 


E/0 

OEUS  • 


(A-1) 


f.  Compute  Base  Facility  load  (BFL) ,  the  number  of  evaluations 

required  to  put  all  relevant  base  organizations  (N  )  through 
one  evaluation  cycle: 


BFL  =  N  X  OE, 
org 


(A-2) 


where  N 

org 


is  the  number  of  base  organizations 
scheduled  to  undergo  evaluation. 


g.  Compute  Estimated  Facility  Load  (EFL) ,  the  expected  number 
of  evaluation  cycles  to  process  all  base  organizations  on 
all  evaluations: 


EFL  =  BFL  X  AEF.  (A-3) 

h.  Determine  the  expected  number  of  re-evaluation  (RE)  cycles: 

RE  =  PLE  X  EFL,  (A-4) 

where  PLE  is  the  estimated  percentage  of  lost 
evaluations . 


i.  Finally,  compute  AFL,  the  expected  number  of  evaluation  cycles 
required  to  process  all  evaluatees,  plus  expected  re- 
evaluacions  : 

AFL  =  RE  +  EFL.  (A-5) 


Step  2.  Compute  Device  Capability  (DC) 

a.  Determine  Work  Segment  Length  (WSL) ,  the  length  of  time  ir 
hours  between  work  starts  and  stops  during  the  work  day. 

b.  Specify  the  Number  of  Work  Seg...ents  (NWS),  the  number  of  WSL 
periods  per  work  day. 
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Estimate  Evaluation  Cycle  Length  (ECL) ,  the  anticipated  time 
in  hours  required  to  conduct  one  evaluation  cycle  using  the 
device. 


Estimate  Time  Between  Cycles  (TBC) ,  time  in  hours  spent  in 
orientation  briefing,  data  collection,  or  critique. 

Compute  the  number  of  Evaluation  Cycles  per  Day  (EC/D) 
(rounded  to  nearest  Integer): 


EC/D  = 


WSL  X  SN 
ECL  +  TBC 


f.  Estimate  Time  Device  Available  (TDA) ,  the  number  of  training 
days  per  year  the  device  Is  expected  to  be  operational  (round 
to  nearest  day) : 

TDA  =  PTA  X  TDY,  (A 

where  PTA  is  the  percentage  of  time  the  device 

is  expected  to  be  available, 

and  TDY  is  the  number  of  training  days  per  year. 

g.  Compute  Device  Capability  (DC),  the  expected  number  of  evalua 
tion  cycles  per  year  that  one  device  will  accommodate: 

DC  =  TDA  X' (EC/D).  (A 


Step  3.  Compute  the  Number  of  Devices  Required  (NDR)  : 


Step  4.  Determine  Incremental  Acquisition  Cost  of  Device  (ACD) 


Compute  Device  Hardware  Cost  (DllC)  ,  the  cost  of  acquiring  the 
required  number  of  devices  (above  the  number  required  solely 
for  training) : 

DHC  =  (NDR  -  NDOH)  x  UCD,  (A 


where  NDOH  is  the  number  of  devices  on  hand  for 
training  purposes, 

and  UCD  is  the  unit  cost  of  the  device. 


b.  Determine  Direct  Personnel  Costs  (DPC).  DPC  is  the  sum  of  the 
estimated  number  by  proposed  pay  grade  of  operators,  evaluators, 
and  maintenance  personnel  multiplied  by  their  salaries  and 
benefits.  Again,  this  is  the  incremental  cost  associated  with 
providing  personnel  for  purely  evaluation  purposes. 

c.  Determine  evaluation-related  Personnel  Training  Costs  (PTC): 

PTC  =  CIT  +  EFT,  (A-11) 

where  CIT  is  the  cost  of  initial  training  for 
operators,  evaluators,  and  maintainers; 

and  EFT  is  the  expected  cost  of  follow-on 
training  (if  any)  plus  the  cost  of  training 
replacements  for  the  Initial  cadre. 

d.  Compute  Administrative  Operating  Costs  (AOC) : 

AOC  =  EXS  4-  TC,  (A-12) 

where  EXS  is  the  cost  of  data  recording  supplies, 
office  supplies,  facilities  supplies,  equipment 
supplies,  utilities,  ammunition,  targets,  and  so 
forth  associated  with  evaluation  activities, 

and  TC  is  the  cost  of  any  required  travel  for 
evaluatees  or  cadre  (again,  evaluation  related) . 

e.  Estimate  Facilities  Cost  (FC) .  FC  is  the  total  cost  of 
required  acquisition,  construction,  or  modification  for 
device-related  facilities  used  solely  for  evaluation. 

f.  Estimate  Annual  Maintenance  Cost  (AMC),  the  annual  cost  of 
maintaining  device-related  facilities  used  solely  for 
eval nation . 

g.  Compute  ACD : 

ACD  =  DHC  +  (DPC  +  AOC  +  AMC)  x  EY  +  PTC  +  FC,  (A-13) 
where  EY  is  the  expected  life  of  the  device. 
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Step  5.  Determine  Total  Device  Cost  (TDC)  : 

TDC  =  ACD  +  EEIR  x  DHC  x  EY, 

where  EEIR  is  the  expected  replacement  rate 
(percentage)  per  year  for  end  item  equipment 
used  for  evaluation  purposes. 


Sten  6.  Determine  Total  Estimated  Cost  of  Alternative  (TECA) : 


After  the  estimated  cost  of  each  device  constituting  a  DORAC  alterna 
tive  is  calculated,  these  costs  are  summed  to  produce  the  Total  Estimated 
Cost  of  the  Alternative  (TECA): 


NDEV 

TECA  =  S  TDC., 
i=l  ^ 

where  NDEV  is  the  number  of  individual  devices 
constituting  DORAC  alternative 


(A-15 


and  TDC.  is  the  estimated  cost  of  the  i 
(from  A-14) . 


device 


The  application  of  (A-15)  in  determining  the  total  estimated  cost  of 
alternative  ^  makes  the  assumption  that  all  individual  device  costs  are 
mutually  exclusive;  that  is,  no  shared  costs  (e.g.,  common  operators, 
evaluators,  facilities,  etc.)  are  involved.  In  the  event  that  shared 
costs  are  involved,  assign  the  overlapping  costs  to  one  device.  In  this 
manner,  shared  costs  are  only  counted  one  time. 
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